00:07:14  * fredkquit (Quit: Leaving.)
00:16:53  * nfitchquit (Quit: Leaving.)
00:21:25  * daviddiasquit (Remote host closed the connection)
00:33:53  * paulfryzeljoined
00:34:36  * chorrelljoined
00:35:39  * paulfryz_joined
00:37:59  * paulfryzelquit (Ping timeout: 240 seconds)
00:39:59  * paulfryz_quit (Ping timeout: 240 seconds)
00:50:11  * marsellquit (Quit: marsell)
00:55:29  * daviddiasjoined
00:58:42  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
01:03:31  * yunongquit (Read error: Connection reset by peer)
01:20:00  * ed209quit (Remote host closed the connection)
01:20:10  * ed209joined
01:23:06  * dap_quit (Quit: Leaving.)
01:26:40  * seldoquit (Remote host closed the connection)
01:36:23  * paulfryzeljoined
01:37:34  * abraxasjoined
01:40:29  * paulfryzelquit (Ping timeout: 240 seconds)
01:42:31  * abraxasquit (Ping timeout: 264 seconds)
01:42:57  * abraxasjoined
01:45:11  * daviddiasquit (Remote host closed the connection)
01:46:22  * saxbyquit (Remote host closed the connection)
01:46:58  * saxbyjoined
01:51:31  * saxbyquit (Ping timeout: 264 seconds)
02:07:11  * seldojoined
02:07:20  * abraxasquit (Remote host closed the connection)
02:13:55  * bixu_joined
02:16:49  * bixuquit (Ping timeout: 240 seconds)
02:18:33  * daviddiasjoined
02:27:52  * daviddiasquit (Read error: Connection reset by peer)
02:35:41  * abraxasjoined
02:37:13  * paulfryzeljoined
02:41:28  * paulfryzelquit (Ping timeout: 246 seconds)
02:50:09  * daviddiasjoined
02:57:15  * AvianFluquit (Remote host closed the connection)
03:19:42  * nfitchjoined
03:24:18  * mcavagequit (Remote host closed the connection)
03:25:15  * mcavagejoined
03:42:25  * daviddiasquit (Remote host closed the connection)
03:52:30  <isaacs>Good evening mantans
03:52:36  <isaacs>Seeing a lot of this: InternalError: an unexpected error occurred
03:53:46  <mcavage>isaacs: on read or wtite
03:53:49  <mcavage>*write?
03:53:54  <isaacs>mcavage: write
03:54:11  <mcavage>hmm. failure mode not graceful, but this what i mailed you about.
03:54:27  <isaacs>oh, ok
03:54:52  <isaacs>mcavage: should i disable the skimdb worker job, and just let it pile up in the db for a bit?
03:54:52  <mcavage>reads are fine right?
03:54:56  <mcavage>(just checking)
03:54:56  <isaacs>seem to be
03:55:00  <mcavage>they should be.
03:55:05  <mcavage>yes, i would just table it for now.
03:55:07  <isaacs>not seeing a proliferation of 503s etc
03:55:12  <isaacs>how long should i table it for?
03:55:17  <mcavage>there are 3 "blips" -- 1 is done, the second is running
03:55:28  <mcavage>in testing on prod data it was 4m per, but the first took 10m
03:55:36  <mcavage>so i'd say if you can suspend for 30m we should be all good.
03:55:41  <isaacs>okie dokie
03:55:44  <isaacs>thanks
03:56:13  <mcavage>sorry :\
03:56:18  <mcavage>this isn't a normal thing.
03:56:48  <isaacs>yeah, i remember the email.
03:56:50  <isaacs>no worries
03:57:06  <isaacs>it actually does work fine if the tgz is in the database as an attachment.
03:57:16  <isaacs>just, it works BETTER if it's in manta instead.
03:57:35  <isaacs>but we're in a low-write time, so it'll catch up fine if i restart in 30 minutes
03:58:02  <mcavage>yeah right now is our slack time before EU picks up. We tried to make this be the least impacted.
03:58:58  <isaacs>ye
03:59:00  <isaacs>yep
04:01:23  * trentmjoined
04:02:43  * trentmpart
04:03:54  <isaacs>mcavage: did actually see this on a read just now
04:03:55  <isaacs>mfind: InternalError: an unexpected error occurred
04:04:36  <mcavage>isaacs: i'm going to *Guess* that it was a timeout because the database is locked up. if you can repro it (likely) with -v I can look server side.
04:05:05  <mcavage>although, mfind seems to be working for me across a very large number of directories.
04:05:16  <isaacs>what is "very large"?
04:05:37  <isaacs>say, 63,000 with around 300,000 files total?
04:05:51  <isaacs>probably it's just not a great time to be asking manta how many tarballs there are :)
04:08:04  <mcavage>i have a tree of lots of dirs/files -- O(50k)
04:10:46  * ringzerowaves hi
04:28:02  * therealkoopaquit (Remote host closed the connection)
04:29:50  * bsdgurujoined
04:31:56  * abraxasquit (Remote host closed the connection)
04:32:32  * abraxasjoined
04:33:06  * ringzeroquit
04:33:22  * ringzerojoined
04:34:06  * ringzeroquit (Client Quit)
04:34:31  * ringzerojoined
04:35:02  * ringzeroquit (Client Quit)
04:35:26  * ringzerojoined
04:35:33  <mcavage>isaacs: I'm running a few more tests, but you should be good to resume now.
04:36:49  * abraxasquit (Ping timeout: 240 seconds)
04:38:35  * paulfryzeljoined
04:40:52  * yunongjoined
04:42:49  * paulfryzelquit (Ping timeout: 240 seconds)
04:44:30  * nfitchquit (Quit: Leaving.)
04:45:37  * mcavagequit (Remote host closed the connection)
04:45:45  * yunongquit (Client Quit)
04:53:04  * bsdguruquit (Quit: bsdguru)
04:53:10  * saxbyjoined
05:17:15  * bsdgurujoined
05:20:42  * abraxasjoined
05:22:08  * papertigersjoined
05:23:26  * papertigersquit (Client Quit)
05:26:29  * abraxasquit (Remote host closed the connection)
05:29:09  * papertigersjoined
05:39:29  * paulfryzeljoined
05:43:29  * paulfryzelquit (Ping timeout: 240 seconds)
06:10:03  * marselljoined
06:20:34  * nfitchjoined
06:24:59  * nfitchquit (Ping timeout: 240 seconds)
06:40:20  * paulfryzeljoined
06:44:29  * paulfryzelquit (Ping timeout: 240 seconds)
06:50:19  * saxbyquit (Remote host closed the connection)
06:50:58  * saxbyjoined
06:51:53  * abraxasjoined
06:55:43  * saxbyquit (Ping timeout: 264 seconds)
06:55:57  * bsdguruquit (Quit: bsdguru)
06:57:28  * bsdgurujoined
07:03:33  * mamashjoined
07:06:35  * abraxasquit (Remote host closed the connection)
07:07:11  * abraxasjoined
07:11:19  * abraxasquit (Ping timeout: 246 seconds)
07:19:07  * marsellquit (Ping timeout: 264 seconds)
07:23:54  * ghostbar_quit (Remote host closed the connection)
07:47:17  * mamashpart
07:47:23  * mamashjoined
07:58:32  * abraxasjoined
08:09:19  * nfitchjoined
08:13:29  * ringzerochanged nick to r1ngzer0
08:14:19  * nfitchquit (Ping timeout: 264 seconds)
08:17:19  * r1ngzer0changed nick to ringzero
08:38:18  * nfitchjoined
08:41:38  * paulfryzeljoined
08:42:25  * nfitchquit (Ping timeout: 240 seconds)
08:45:59  * paulfryzelquit (Ping timeout: 240 seconds)
08:51:37  * bsdguruquit (Quit: bsdguru)
09:11:34  * ringzerochanged nick to r1ngzer0
09:16:01  * abraxasquit (Remote host closed the connection)
09:22:30  * bsdgurujoined
09:23:00  * ghostbarjoined
09:23:24  * ins0mniajoined
09:31:51  * abraxasjoined
09:42:23  * paulfryzeljoined
09:46:59  * paulfryzelquit (Ping timeout: 240 seconds)
10:06:42  * abraxasquit (Remote host closed the connection)
10:07:16  * abraxasjoined
10:11:55  * abraxasquit (Ping timeout: 264 seconds)
10:13:33  * ghostbarquit (Remote host closed the connection)
10:25:39  * mariuspjoined
10:26:06  * bixujoined
10:27:05  * nfitchjoined
10:27:10  * bixu__joined
10:27:13  * bixu_quit (Read error: Connection reset by peer)
10:28:54  * bixu__quit (Remote host closed the connection)
10:29:12  * bixu_joined
10:30:29  * bixuquit (Ping timeout: 241 seconds)
10:31:32  * nfitchquit (Ping timeout: 246 seconds)
10:40:41  * therealkoopajoined
10:43:10  * paulfryzeljoined
10:44:29  * bixu_quit (Ping timeout: 240 seconds)
10:47:59  * paulfryzelquit (Ping timeout: 269 seconds)
10:52:25  * bixujoined
10:54:26  * bixuquit (Remote host closed the connection)
10:54:56  * bixujoined
10:56:33  * bixu_joined
11:00:01  * bixuquit (Ping timeout: 240 seconds)
11:00:06  * bixu_quit (Remote host closed the connection)
11:00:35  * bixujoined
11:04:00  * bixu_joined
11:06:59  * bixuquit (Ping timeout: 240 seconds)
11:20:02  * therealkoopaquit (Remote host closed the connection)
11:33:04  * mariuspquit
11:33:45  * bixu_quit (Remote host closed the connection)
11:37:48  * abraxasjoined
11:44:10  * paulfryzeljoined
11:48:29  * paulfryzelquit (Ping timeout: 240 seconds)
11:49:06  * saxbyjoined
11:49:43  * bixujoined
11:53:29  * saxbyquit (Ping timeout: 240 seconds)
11:55:56  * mamashpart
12:10:30  * abraxasquit (Remote host closed the connection)
12:11:07  * abraxasjoined
12:15:50  * mamashjoined
12:15:51  * nfitchjoined
12:16:07  * abraxasquit (Ping timeout: 264 seconds)
12:19:42  * bixuquit (Remote host closed the connection)
12:20:01  * nfitchquit (Ping timeout: 240 seconds)
12:28:52  * bixujoined
12:29:17  * bixuquit (Read error: Connection reset by peer)
12:29:41  * bixujoined
12:30:33  * chorrelljoined
12:30:38  * chorrellchanged nick to chorrell-away
12:34:01  * bixuquit (Ping timeout: 246 seconds)
12:43:45  * marselljoined
12:44:55  * paulfryzeljoined
12:48:59  * paulfryzelquit (Ping timeout: 240 seconds)
12:52:09  * bixujoined
12:56:29  * bixuquit (Ping timeout: 240 seconds)
13:00:57  * chorrell-awayquit (Quit: My Mac has gone to sleep. ZZZzzz…)
13:10:52  * chorrelljoined
13:12:47  * AvianFlujoined
13:33:12  * therealkoopajoined
13:36:09  * bixujoined
13:50:09  * AvianFluquit (Remote host closed the connection)
13:50:19  * saxbyjoined
13:55:07  * saxbyquit (Ping timeout: 264 seconds)
14:04:36  * nfitchjoined
14:09:31  * nfitchquit (Ping timeout: 264 seconds)
14:11:46  * abraxasjoined
14:13:28  * chorrellchanged nick to chorrell-away
14:15:59  * abraxasquit (Ping timeout: 241 seconds)
14:22:29  * saxbyjoined
14:26:55  * saxbyquit (Ping timeout: 264 seconds)
14:33:53  * chorrell-awaychanged nick to chorrell
14:36:21  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
14:37:18  * mcavagejoined
14:41:59  * mamashpart
15:04:09  * paulfryz_joined
15:05:18  * therealk_joined
15:05:47  * therealkoopaquit (Write error: Connection reset by peer)
15:06:02  * therealk_quit (Read error: No route to host)
15:07:29  * therealkoopajoined
15:07:34  * therealkoopaquit (Remote host closed the connection)
15:09:19  * therealkoopajoined
15:09:56  * nfitchjoined
15:10:31  * therealkoopaquit (Client Quit)
15:15:23  * therealkoopajoined
15:23:17  * saxbyjoined
15:27:53  * saxbyquit (Ping timeout: 255 seconds)
15:46:40  * AvianFlujoined
15:56:11  * fredkjoined
16:10:49  * mamashjoined
16:12:33  * abraxasjoined
16:16:49  * abraxasquit (Ping timeout: 240 seconds)
16:18:26  * ira_joined
16:25:21  * yunongjoined
16:30:02  * seldoquit (Remote host closed the connection)
16:30:25  * r1ngzer0changed nick to ringzero
16:33:42  * dap_joined
17:10:23  * seldojoined
17:11:57  * saxbyjoined
17:14:42  * yunongquit (Quit: Leaving.)
17:17:10  * therealkoopaquit (Remote host closed the connection)
17:18:56  * saxbyquit (Remote host closed the connection)
17:25:03  * ira_changed nick to ira
17:31:07  * therealkoopajoined
17:31:30  * saxbyjoined
17:36:19  * therealkoopaquit (Ping timeout: 287 seconds)
17:37:04  * therealkoopajoined
17:38:01  * iraquit (Ping timeout: 240 seconds)
17:38:05  * saxbyquit (Remote host closed the connection)
17:39:05  * mcavagequit (Remote host closed the connection)
17:40:06  * saxbyjoined
17:40:22  * mcavagejoined
17:45:01  * daviddiasjoined
18:00:21  * yunongjoined
18:02:35  * saxbyquit (Remote host closed the connection)
18:06:50  * bsdguruquit (Quit: bsdguru)
18:07:29  * daviddia_joined
18:07:29  * daviddiasquit (Read error: Connection reset by peer)
18:09:58  * mcavagequit (Remote host closed the connection)
18:10:54  * mcavagejoined
18:11:07  * mcavagequit (Remote host closed the connection)
18:11:14  * mcavagejoined
18:11:20  * saxbyjoined
18:13:24  * abraxasjoined
18:16:11  * yunong1joined
18:18:01  * abraxasquit (Ping timeout: 240 seconds)
18:20:55  * yunong1quit (Ping timeout: 264 seconds)
18:21:33  * dap_quit (Quit: Leaving.)
18:46:59  * marsellquit (Ping timeout: 240 seconds)
18:49:06  * saxbyquit (Remote host closed the connection)
18:56:21  * marselljoined
19:13:06  * ringzerochanged nick to r1ngzer0
19:17:29  * yunong1joined
19:21:53  * yunong1quit (Ping timeout: 255 seconds)
19:26:41  * notmattquit (Remote host closed the connection)
19:27:14  * notmattjoined
19:31:37  * notmattquit (Ping timeout: 240 seconds)
19:31:44  * r1ngzer0changed nick to ringzero
19:48:17  * yunong1joined
19:52:43  * yunong1quit (Ping timeout: 264 seconds)
19:53:07  * paulfryz_quit (Remote host closed the connection)
19:55:47  * bahamas10quit (Read error: Connection reset by peer)
19:59:08  * saxbyjoined
20:14:29  * abraxasjoined
20:15:00  * saxbyquit (Remote host closed the connection)
20:15:38  * saxbyjoined
20:18:49  * abraxasquit (Ping timeout: 240 seconds)
20:19:09  * yunong1joined
20:19:59  * saxbyquit (Ping timeout: 240 seconds)
20:23:29  * yunong1quit (Ping timeout: 240 seconds)
20:23:51  * paulfryzeljoined
20:40:09  * ghostbarjoined
21:10:40  * mamashpart
21:16:24  * yunong1joined
21:20:29  * yunong1quit (Ping timeout: 240 seconds)
21:39:40  * therealkoopaquit (Remote host closed the connection)
21:46:23  * saxbyjoined
21:50:34  * notmattjoined
21:53:36  * saxbyquit (Remote host closed the connection)
21:55:49  * paulfryzelquit (Remote host closed the connection)
21:56:34  * therealkoopajoined
22:16:34  * yunong1joined
22:16:52  * saxbyjoined
22:20:49  * yunong1quit (Ping timeout: 240 seconds)
22:22:32  * mcavagequit (Remote host closed the connection)
22:26:52  * paulfryzeljoined
22:26:55  * saxbyquit (Remote host closed the connection)
22:28:12  * paulfryzelquit (Read error: Connection reset by peer)
22:28:42  * paulfryzeljoined
22:28:49  * ed209quit (Ping timeout: 240 seconds)
22:29:23  * nfitchquit (Quit: Leaving.)
22:33:59  * paulfryzelquit (Ping timeout: 240 seconds)
22:50:56  * ringzeroquit (Read error: Connection reset by peer)
22:51:02  * ringzero_joined
22:52:21  * ringzerojoined
22:55:29  * ringzero_quit (Ping timeout: 240 seconds)
22:58:32  * saxbyjoined
22:59:56  * nfitchjoined
23:04:38  * nfitchquit (Ping timeout: 255 seconds)
23:10:49  * ins0mniaquit (Ping timeout: 240 seconds)
23:13:01  * ghostbarquit (Remote host closed the connection)
23:14:47  * ins0mniajoined
23:17:16  * yunong1joined
23:18:12  * bahamas10joined
23:21:00  * yunong1quit (Ping timeout: 267 seconds)
23:35:51  * mcavagejoined
23:37:04  * ringzero_joined
23:37:32  * therealk_joined
23:40:22  * ghostbarjoined
23:42:55  * ghostbar_joined
23:44:25  * ghostbarquit (Ping timeout: 240 seconds)
23:45:43  * ringzeroquit (Ping timeout: 321 seconds)
23:46:09  * therealkoopaquit (Ping timeout: 242 seconds)
23:46:12  * notmattquit (Remote host closed the connection)
23:46:26  * notmattjoined