00:02:35  * _Tenchi_joined
00:16:13  * paulfryz_joined
00:20:33  * paulfryz_quit (Ping timeout: 245 seconds)
00:33:39  * daviddia_quit (Remote host closed the connection)
00:34:19  * daviddiasjoined
00:36:54  * daviddiasquit (Read error: No route to host)
00:37:20  * daviddiasjoined
01:05:31  * AvianFluquit (Ping timeout: 252 seconds)
01:08:41  * fredkquit (Quit: Leaving.)
01:09:12  * fredkjoined
01:09:28  * fredkquit (Client Quit)
01:16:56  * paulfryz_joined
01:21:23  * paulfryz_quit (Ping timeout: 245 seconds)
01:21:31  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
01:25:48  * lloyddequit (Remote host closed the connection)
01:55:04  * _Tenchi_quit (Quit: Leaving)
01:56:33  * lloyddejoined
02:00:58  * lloyddequit (Ping timeout: 245 seconds)
02:09:28  * yunongquit (Quit: Leaving.)
02:09:45  * fredkjoined
02:17:38  * paulfryz_joined
02:18:15  * fredkquit (Ping timeout: 240 seconds)
02:19:06  * AvianFlujoined
02:21:48  * paulfryz_quit (Ping timeout: 245 seconds)
02:32:23  * daviddiasquit
02:35:12  * ryancnelsonpart
02:36:28  * dap_quit (Quit: Leaving.)
02:47:54  * AvianFluquit (Read error: Connection reset by peer)
02:48:33  * AvianFlujoined
02:57:36  * lloyddejoined
03:01:57  * lloyddequit (Ping timeout: 246 seconds)
03:04:46  * abraxasjoined
03:08:56  * abraxasquit (Ping timeout: 240 seconds)
03:18:28  * paulfryz_joined
03:22:38  * paulfryz_quit (Ping timeout: 245 seconds)
03:56:00  * hij1nxquit (Quit: WeeChat 0.3.2)
03:58:13  * lloyddejoined
04:02:38  * lloyddequit (Ping timeout: 245 seconds)
04:02:56  * AvianFluquit (Read error: Connection reset by peer)
04:03:33  * AvianFlujoined
04:15:06  * fredkjoined
04:15:44  * nfitchquit (Quit: Leaving.)
04:19:07  * paulfryzeljoined
04:19:39  * fredkquit (Ping timeout: 272 seconds)
04:23:31  * paulfryzelquit (Ping timeout: 252 seconds)
04:45:15  * fredkjoined
04:49:43  * fredkquit (Ping timeout: 245 seconds)
04:58:57  * lloyddejoined
05:03:03  * lloyddequit (Ping timeout: 245 seconds)
05:09:29  * nfitchjoined
05:13:40  * nfitchquit (Ping timeout: 240 seconds)
05:20:01  * paulfryzeljoined
05:24:18  * paulfryzelquit (Ping timeout: 245 seconds)
05:30:19  * lloyddejoined
05:30:53  * lloydde_joined
05:35:02  * lloyddequit (Ping timeout: 264 seconds)
06:06:07  * AvianFluquit (Remote host closed the connection)
06:20:48  * paulfryzeljoined
06:25:08  * paulfryzelquit (Ping timeout: 245 seconds)
06:58:13  * nfitchjoined
07:02:30  * nfitchquit (Ping timeout: 245 seconds)
07:06:35  * abraxasjoined
07:11:23  * abraxasquit (Ping timeout: 272 seconds)
07:13:56  * nfitchjoined
07:18:04  * nfitchquit (Ping timeout: 240 seconds)
07:21:30  * paulfryzeljoined
07:24:30  * lloydde_quit (Remote host closed the connection)
07:25:58  * paulfryzelquit (Ping timeout: 245 seconds)
07:55:15  * lloyddejoined
07:59:47  * lloyddequit (Ping timeout: 272 seconds)
08:01:37  * marsellquit (Quit: marsell)
08:11:20  * mariuspjoined
08:22:16  * paulfryzeljoined
08:26:23  * paulfryzelquit (Ping timeout: 245 seconds)
08:55:28  * lloyddejoined
08:57:05  * lloyddequit (Read error: Connection reset by peer)
08:57:25  * lloyddejoined
09:02:08  * lloyddequit (Ping timeout: 260 seconds)
09:02:42  * nfitchjoined
09:04:33  * nfitch1joined
09:04:33  * nfitchquit (Read error: Connection reset by peer)
09:07:28  * abraxasjoined
09:08:53  * nfitch1quit (Ping timeout: 245 seconds)
09:12:21  * abraxasquit (Ping timeout: 272 seconds)
09:23:07  * paulfryzeljoined
09:27:13  * paulfryzelquit (Ping timeout: 245 seconds)
09:58:08  * lloyddejoined
10:03:01  * lloyddequit (Ping timeout: 272 seconds)
10:23:52  * paulfryzeljoined
10:28:03  * paulfryzelquit (Ping timeout: 245 seconds)
10:53:16  * nfitchjoined
10:58:24  * nfitchquit (Ping timeout: 272 seconds)
10:58:54  * lloyddejoined
11:01:09  * marselljoined
11:03:15  * lloyddequit (Ping timeout: 246 seconds)
11:08:17  * abraxasjoined
11:13:13  * abraxasquit (Ping timeout: 272 seconds)
11:24:34  * paulfryzeljoined
11:28:53  * paulfryzelquit (Ping timeout: 245 seconds)
11:59:40  * lloyddejoined
12:03:56  * lloyddequit (Ping timeout: 245 seconds)
12:25:21  * paulfryzeljoined
12:29:43  * paulfryzelquit (Ping timeout: 245 seconds)
12:42:00  * nfitchjoined
12:46:36  * nfitchquit (Ping timeout: 265 seconds)
13:00:38  * lloyddejoined
13:04:46  * lloyddequit (Ping timeout: 245 seconds)
13:09:15  * abraxasjoined
13:13:55  * abraxasquit (Ping timeout: 272 seconds)
13:26:07  * paulfryzeljoined
13:30:33  * paulfryzelquit (Ping timeout: 245 seconds)
14:01:19  * lloyddejoined
14:01:47  * AvianFlujoined
14:05:33  * lloyddequit (Ping timeout: 245 seconds)
14:07:08  * AvianFlu_joined
14:08:17  * AvianFluquit (Ping timeout: 265 seconds)
14:26:53  * paulfryzeljoined
14:30:46  * nfitchjoined
14:31:23  * paulfryzelquit (Ping timeout: 245 seconds)
14:35:21  * nfitchquit (Ping timeout: 265 seconds)
15:02:04  * lloyddejoined
15:07:04  * lloyddequit (Ping timeout: 264 seconds)
15:20:46  * lloyddejoined
15:21:01  * lloyddequit (Read error: Connection reset by peer)
15:21:26  * lloyddejoined
15:27:39  * paulfryzeljoined
15:31:48  * paulfryzelquit (Ping timeout: 245 seconds)
15:43:59  * papajuansjoined
16:17:44  * lloyddequit (Remote host closed the connection)
16:19:35  * nfitchjoined
16:24:17  * nfitchquit (Ping timeout: 272 seconds)
16:28:24  * paulfryzeljoined
16:31:11  * mamashjoined
16:32:38  * paulfryzelquit (Ping timeout: 245 seconds)
16:56:25  * papajuan_joined
16:56:25  * papajuansquit (Read error: Connection reset by peer)
17:24:19  * lloyddejoined
17:29:08  * paulfryzeljoined
17:33:28  * paulfryzelquit (Ping timeout: 245 seconds)
17:33:33  * AvianFlu_quit (Read error: Connection reset by peer)
17:41:10  * mamashpart
17:49:11  * mamashjoined
17:52:26  * AvianFlujoined
18:00:57  * nfitchjoined
18:04:17  * mamashpart
18:04:38  * mamashjoined
18:07:21  <isaacs>Hm. Seems like manta.ftw is kinda slow.
18:07:23  <isaacs>that's unfortunate.
18:07:37  <isaacs>i'm making much fewer HTTP requests now, but it's actually going slower than before.
18:07:41  <tjfontaine>well, it's a lot of http requests
18:07:44  <tjfontaine>hm
18:10:17  * lloyddequit (Remote host closed the connection)
18:12:30  <rmustacc>Are the requests maybe getting serialized in a different way?
18:22:01  <isaacs>Doing client.ftw() and matching based on length is about 30% slower than calling client.info() on each remote path
18:22:04  <isaacs>seems weird, since it’s way fewer http calls
18:22:11  <isaacs>(copied from joyent internal jabber [email protected] channel)
18:23:30  <isaacs>i don't know if there'd be some tipping point where ftw is faster, or whatever.
18:24:02  <tjfontaine>it would be interesting to see the latencies for the http calls themselves
18:24:06  <isaacs>tjfontaine: this is while testing with a local folder of 12 files. 6 at the first level, 6 in a subdir
18:24:31  <isaacs>i can print some debuggy timing stuff.
18:24:54  <isaacs>but i'm finding that my refactor of manta-sync with cuttlefish is significantly slower for this small number of files.
18:25:02  <isaacs>still pretty fast, but that's a big loss.
18:25:13  <isaacs>and the local js stuff is a rounding error, afaict
18:26:34  <tjfontaine>which types of requests does ftw/find send vs info? info is just a GET on the directory?
18:27:57  <isaacs>ftw is sending client.ls calls
18:28:59  <isaacs>well, it does an info on the root, and then recursive ls over the tree
18:29:54  * paulfryzeljoined
18:30:08  <isaacs>hrm. I guess I thought that ftw was doing some serverside magic.
18:30:11  <isaacs>streaming json or something
18:30:32  <isaacs>serves me right for not RTFS :)
18:30:45  <tjfontaine>:)
18:31:18  <isaacs>still, shoudl be just 2 ls() calls in this case.
18:31:51  <isaacs>vs a dozen info() calls
18:32:07  <isaacs>and then a ftw() *anyway* to do the deletes.
18:34:18  * paulfryzelquit (Ping timeout: 245 seconds)
18:39:18  * papajuan_quit (Remote host closed the connection)
18:40:18  * papajuansjoined
18:41:58  * mamashpart
19:03:47  <isaacs>ahhhh.... i see.
19:04:07  <isaacs>if you have a lot of files there, and aren't doing deletes, then the existing manta-sync doesn't do an ftw at all.
19:04:22  <isaacs>it'd be nice if ftw() had a filter or something
19:04:38  <isaacs>so i could avoid traversing dirs that i don't have anyway
19:08:37  <isaacs>tjfontaine: figured it out. if i do --delete, or if the remote doesn't have a lot of files already, then cuttlefish is faster.
19:09:34  <tjfontaine>interesting.
19:10:41  <isaacs>if ftw has a filter, then cuttlefish is always faster.
19:10:56  <isaacs>especially if there's a bigass dir in there that i'll be deleting anyway
19:11:30  <isaacs>but, if you're adding just a few files, to a folder with LOTS of files, and not deleting them, then just doing info() on those few is way faster.
19:11:38  * nfitchquit (Quit: Leaving.)
19:12:31  * nfitchjoined
19:12:35  * nfitchquit (Client Quit)
19:13:40  * nfitchjoined
19:13:52  * nfitchquit (Client Quit)
19:14:12  * nfitchjoined
19:14:21  * nfitchquit (Client Quit)
19:18:39  * AvianFluquit (Read error: Connection reset by peer)
19:30:41  * paulfryzeljoined
19:35:08  * paulfryzelquit (Ping timeout: 245 seconds)
20:09:24  * mamashjoined
20:31:28  * paulfryzeljoined
20:35:58  * paulfryzelquit (Ping timeout: 245 seconds)
20:42:18  * mamashpart
20:55:10  * mariuspquit (Remote host closed the connection)
21:00:46  * mariuspjoined
21:01:00  * nfitchjoined
21:05:08  * nfitchquit (Ping timeout: 245 seconds)
21:11:23  * mariuspquit (Remote host closed the connection)
21:24:40  <isaacs>Hm. It's not depth.
21:24:47  <isaacs>it's just mls taking a long ass time.
21:24:48  <isaacs>https://gist.github.com/isaacs/7620148
21:25:36  <isaacs>-vv shows x-response-time's of 14, 18ms or so
21:25:42  <isaacs>so something's not quite right
21:27:07  <isaacs>adding some timing instrumentation in cuttlefish shows this: https://gist.github.com/isaacs/7620167
21:27:25  <isaacs>note the 0.89579s from calling client.ls until the ls cb being invoked.
21:27:36  <isaacs>and then another 400ms before the first entry
21:29:54  <isaacs>wow, _no_dir_check makes it WAY faster
21:30:46  <isaacs>with _no_dir_check, it looks like this: https://gist.github.com/isaacs/7620191
21:32:10  * paulfryzeljoined
21:36:23  * paulfryzelquit (Ping timeout: 245 seconds)
22:25:57  * mariuspjoined
22:26:25  * mariuspquit (Remote host closed the connection)
22:33:01  * paulfryzeljoined
22:35:34  * daviddiasjoined
22:37:13  * paulfryzelquit (Ping timeout: 245 seconds)
22:38:22  * lloyddejoined
22:49:59  * nfitchjoined
22:54:38  * nfitchquit (Ping timeout: 265 seconds)
23:28:31  * papajuansquit (Remote host closed the connection)
23:32:25  * daviddiasquit
23:33:55  * paulfryzeljoined
23:38:03  * paulfryzelquit (Ping timeout: 245 seconds)