00:09:21  * mikealjoined
00:10:27  * wolfeidaujoined
00:11:34  * wolfeida_joined
00:15:29  * wolfeidauquit (Ping timeout: 265 seconds)
00:19:27  * mikealquit (Ping timeout: 272 seconds)
00:32:50  * wolfeida_quit (Remote host closed the connection)
00:35:22  * wolfeida_joined
00:44:12  <brycebaril>rvagg: is there any way to know if {sync: true} is working? In my batch load testing I can't really see a difference.
00:46:41  <brycebaril>mostly I'm trying to figure out a way to signal backpressure to a writeStream wrapper. I think what I need now is some way to signal back-pressure upstream
00:49:19  <brycebaril>If I avoid objectMode streams & key/value encoding logic with pure binary (multibuffer) streams, I can significantly cut the memory & time, but it appears to hit a point where I'm still ending up queing most/all of it into memory
00:49:31  * julianduquequit (Ping timeout: 246 seconds)
00:49:41  <levelbot>[npm] [email protected] <http://npm.im/level-assoc>: relational foreign key associations (hasMany, belongsTo) for leveldb (@substack)
00:52:22  <substack>row.blah({ limit: n }) works yay
00:52:39  * thlorenzjoined
00:53:04  <brycebaril>I'll try a manual throttle to see if that helps for now, but it would be nice if there was something exposed in the LevelDB api that correlated with write task readiness (or LOG->table pending work)
00:56:51  * thlorenzquit (Ping timeout: 245 seconds)
01:03:57  * fb55joined
01:04:22  * fb55quit (Remote host closed the connection)
01:04:48  * fb55joined
01:09:28  * fb55quit (Ping timeout: 246 seconds)
01:15:44  * mikealjoined
01:18:12  * wolfeida_quit (Remote host closed the connection)
01:20:37  * mikealquit (Ping timeout: 272 seconds)
01:29:07  <brycebaril>rvagg: Hmm... well, duh -- for bulk loading {sync: true} wouldn't matter. Is there anyway to expose a sync batch that *does* behave synchronously WRT Node?
01:30:28  * zz_eugenewarechanged nick to eugeneware
01:35:46  <rvagg>brycebaril: in terms of blocking behaviour you *could* expose a synchronous/blocking batch, but we're not going to do that; ideally the callback should be all you need but what I'm hearing is that it isn't for this bulk load situation
01:37:58  <rvagg>it would be lovely to have an event system for leveldb that gave us insight into compactions and other internal behaviours
01:38:19  <rvagg>not impossible but it would mean properly digging in to the guts of leveldb to expose it, which isn't very friendly for an upgrade path
01:38:24  <rvagg>... but it would be interesting!
01:41:15  * wolfeidaujoined
01:41:45  * esundahljoined
01:46:52  * wolfeidauquit (Remote host closed the connection)
01:47:24  * wolfeidaujoined
01:49:15  * rudjoined
01:49:38  * thlorenzjoined
01:49:51  <rvagg>I'm thinking that we should perhaps make our own Log implementation that can be plugged in to and streamed so you can watch progress and/or pipe it to an actual log somewhere if you like
01:52:17  * wolfeidauquit (Ping timeout: 272 seconds)
02:04:06  * rudquit (Quit: rud)
02:51:00  <nathan7>rvagg: does LevelDB have any dtraceness?
02:53:40  * thlorenzquit (Remote host closed the connection)
02:54:00  <ogd>rvagg: from what i can tell compaction is important for bulk loading stuff but theres no visibility into what the compaction state is
02:54:23  <ogd>rvagg: the callback for a batch fires and compaction might still be happening
02:55:27  <ogd>rvagg: but i think its only an issue if you are writing too much data
02:56:38  <ogd>so i guess the answer is to not write too much data
02:57:06  <ogd>i'm still interested to see someone test out that multiple leveldbs for writing that get combined later idea
02:57:55  * ramitosjoined
03:06:01  * nnnnathannquit (Ping timeout: 245 seconds)
03:12:31  * ramitosquit (Quit: Computer has gone to sleep.)
03:14:16  * ramitosjoined
03:24:49  * thlorenzjoined
03:30:13  * ramitosquit (Quit: Textual IRC Client: www.textualapp.com)
03:32:48  * thlorenzquit (Ping timeout: 240 seconds)
03:38:03  * jcrugzzjoined
03:45:23  * dominictarrjoined
03:51:34  * dominictarrquit (Quit: dominictarr)
03:52:08  * timoxleyjoined
03:52:22  <juliangruber>substack: sweet
04:18:49  * jcrugzzquit (Ping timeout: 246 seconds)
04:29:43  * thlorenzjoined
04:34:25  * thlorenzquit (Ping timeout: 272 seconds)
04:49:07  <rvagg>ogd: https://github.com/rvagg/node-leveldown/pull/67
04:59:11  * jxsonquit (Remote host closed the connection)
05:04:22  <ogd>rvagg: nice
05:12:45  <brycebaril>rvagg: awesome, I'll take a look. Yeah, I don't think the bulk load situation needs much, just some way of propogating backpressure. The issue is with the async callbacks there is nothing to tell upstream to pause
05:15:52  <rvagg>yeah.. using log data for anything useful would be pretty hacky; but the info is interesting and educational nonetheless
05:45:13  <brycebaril>rvagg: what about deferring the batch callback until the sync was done if {sync: true}? That way the api doesn't change, just timing if they request synchronous writes?
05:45:30  <rvagg>brycebaril: that's the way it should happen
05:47:31  * esundahlquit (Remote host closed the connection)
05:47:44  <brycebaril>Oh? Hmm, lemme see if I can see a timing difference between batches with sync true or false. Maybe it is something else.
05:48:39  <levelbot>[npm] [email protected] <http://npm.im/memdown>: An drop-in replacement for LevelDOWN that works in memory only (@rvagg)
05:50:02  * mikealjoined
05:50:06  <brycebaril>I'm still trying to figure out what is taking up heapspace memory in my bulk loader. So if buffers mostly live outside heap, any idea what the heap memory component looks like for them?
05:53:40  <levelbot>[npm] [email protected] <http://npm.im/memdown>: An drop-in replacement for LevelDOWN that works in memory only (@rvagg)
05:59:09  <rvagg>join #libuv and bother trevnorris
05:59:23  <ogd>hmm something im doing in dat is slowing reads down to 7k/s whereas i get ~100k/s with normal levelup... need to investigate
05:59:29  <rvagg>actually, trevnorris is in here
06:05:10  <levelbot>[npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg)
06:07:39  <levelbot>[npm] [email protected] <http://npm.im/level-mem>: A convenience package bundling LevelUP & MemDOWN (@rvagg)
06:09:11  * eugenewarechanged nick to zz_eugeneware
06:19:54  <brycebaril>Yeah, I get the same mean/median timing for batches if sync is true or false
06:22:46  <rvagg>that's odd
06:23:04  <rvagg>chained batch or array batch?
06:23:20  <brycebaril>array
06:24:30  <rvagg>I'll double-check that sync is getting properly passed down to leveldb when I have time later tonight, it's impossible to unit test for unfortunately
06:24:56  <rvagg>can't mock out kernel fs writes unfortunately!
06:28:07  <brycebaril>I'm still getting 28k records per second, which isn't bad by any means
06:29:33  <brycebaril>Apparently memwatch heapdump on a 700+ meg heapspace isn't a good idea
06:29:44  <brycebaril>22449 bryce 20 0 11.6g 6.5g 788 R 26 84.2 5:29.16 node
06:45:26  <brycebaril>ok, solved my memory problem
06:45:41  <brycebaril>array batch = bad for memory
06:47:43  <brycebaril>rvagg: memory use for standard levelup rs.pipe(ws) for a 5 million record database: (1994.csv that ogd uses): 1364895432
06:48:06  <brycebaril>using the buffers-only read/write streams I'm working on: 13701672
06:48:12  <brycebaril>those are JS heap space
06:49:01  <brycebaril>rss 3_153_367_040 vs 554_786_816
06:50:24  <brycebaril>timing 257s -> 182s
06:58:31  <rvagg>this is why writestream has to go, it's a bikeshed
07:02:52  <brycebaril>I'm over worrying about keeping it in the core. I'll still use some sort of writeStream interface, but I know what I'm doing.
07:03:38  <brycebaril>If someone with a big level instance takes the README's advice about making a copy() function they are guaranteed to have a bad experience. That's probably reason enough to pull it.
07:06:34  <rvagg>in fact there's a .copy() exported on LevelUP that probably shouldn't be there
07:06:57  <rvagg>see latest comment on #199
07:07:29  <brycebaril>I'm assuming by bikeshed you mean that any given developer/application is going to see a different need for writeStream that needs a completely different implementation? If so, I completely agree.
07:15:09  * wolfeidaujoined
07:16:03  * zz_eugenewarechanged nick to eugeneware
07:33:15  * eugenewarechanged nick to zz_eugeneware
08:46:03  * fb55joined
08:46:07  * dominictarrjoined
09:07:41  <levelbot>[npm] [email protected] <http://npm.im/level-ws>: A basic WriteStream implementation for LevelUP (@rvagg)
09:07:58  * fb55quit (Remote host closed the connection)
09:08:25  * fb55joined
09:12:49  * fb55quit (Ping timeout: 246 seconds)
09:17:40  <levelbot>[npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg)
09:32:53  * thlorenzjoined
09:33:41  <levelbot>[npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg)
09:35:56  * pgtejoined
09:37:26  * thlorenzquit (Ping timeout: 264 seconds)
09:38:47  * timoxleyquit (Remote host closed the connection)
09:44:46  <rvagg>FYI anyone not following #199 (remove WriteStream) https://github.com/rvagg/node-levelup/issues/199#issuecomment-26194464
09:51:03  * fb55joined
09:56:39  * pgtequit (Remote host closed the connection)
09:57:11  * pgtejoined
10:01:47  * pgtequit (Ping timeout: 265 seconds)
10:16:03  * zz_eugenewarechanged nick to eugeneware
10:17:02  * fb55quit (Ping timeout: 240 seconds)
10:24:53  * fb55joined
11:55:23  * dominictarrquit (Quit: dominictarr)
11:56:52  * dominictarrjoined
11:58:18  * dominictarrquit (Client Quit)
11:59:28  * eugenewarechanged nick to zz_eugeneware
12:19:46  * thlorenzjoined
12:22:09  * jez0990quit (Remote host closed the connection)
12:24:57  * jez0990joined
12:25:37  * thlorenzquit (Ping timeout: 272 seconds)
12:28:54  * jez0990quit (Read error: Connection reset by peer)
12:28:55  * dominictarrjoined
12:31:53  * carolynschanged nick to foxcat
12:31:53  <dominictarr>rvagg: hey, did you document your setup for your nexus7/rpi thing?
12:32:13  * foxcatpart
12:35:19  <hij1nx>i noticed that goatdb has migrated to australia...
12:40:09  <rvagg>dominictarr: no, no documentation, typical linux hackety hack
12:40:29  <rvagg>dominictarr: if you need it I guess I can
12:41:00  <rvagg>hij1nx: where is it?
12:41:08  <dominictarr>rvagg: main question: did you use a seprate router? how do you get the rpi to connect to the right network?
12:41:26  <rvagg>dominictarr: linux wifi tools
12:41:36  <rvagg>dominictarr: pi has a wifi dongle and a bluetooth dongle
12:41:57  <rvagg>the bluetooth is the most important, it's set up so that I can connect to it from the nexus
12:42:04  <dominictarr>so, you connect to the nexus7 via bt?
12:42:08  <rvagg>I can pull out the wifi and it still works, I just get no external networking
12:42:15  <rvagg>yeah, bluetooth from nexus to pi
12:42:21  <rvagg>bluetooth networking to be exact
12:42:27  <dominictarr>right - I see
12:42:32  <rvagg>that was a bit of a pain but I think I could probably document the recipe that I landed on
12:43:13  <rvagg>it's always predictable, the pi shows up with ip address which I start a session on when the bluetooth is configured
12:43:32  <rvagg>then when I'm in I manually configure whatever wifi network is available using wpa_supplicant ... grr ... linux wifi crap
12:43:49  * rudjoined
12:43:49  * rudquit (Changing host)
12:43:49  * rudjoined
12:44:24  <rvagg>ingredients: pi, bluetooth dongle pi compatible, wifi dongle pi compatible, nexus 7, bluetooth keyboard for nexus 7, battery pack for pi and nexus and keyboard where required
12:45:43  <rvagg>the last leg of my trip home I had the pi & battery tucked away in my bag in the overhead baggage locker and it never came out, I just fired up the nexus when I was ready to hack and connected via bluetooth
12:46:25  <rvagg>I do have a problem with repeat bluetooth connections, it doesn't seem to reset well so as insurance if I'm disconnecting the nexus from the pi and I know I'll want to reconnect later I'll tell it to reboot just to start fresh and so I don't have to rip the power out and put it back in again to force a restart
12:46:36  <rvagg>but that's something that just needs tweaking so I can reconnect later
12:46:47  <rvagg>otherwise it's pretty stable and works well
12:46:59  <rvagg>and I actually got a fair bit of work done on the plane this time, more than I've done before
12:49:03  <rvagg>heh, pomke has goatdb now
12:49:47  * rvaggbed
12:56:19  * zz_eugenewarechanged nick to eugeneware
13:23:53  * fb55quit (Remote host closed the connection)
13:24:20  * fb55joined
13:26:41  * rud_joined
13:28:26  * rudquit (Ping timeout: 264 seconds)
13:28:27  * rud_changed nick to rud
13:29:35  * fb55quit (Ping timeout: 272 seconds)
13:34:05  * dominictarrquit (Quit: dominictarr)
13:58:41  * pgtejoined
14:04:51  * dominictarrjoined
14:17:09  * eugenewarechanged nick to zz_eugeneware
14:18:46  * zz_eugenewarechanged nick to eugeneware
14:19:11  * pgtequit (Remote host closed the connection)
14:24:30  * pgtejoined
14:25:17  * pgtequit (Remote host closed the connection)
14:29:33  * kenansulaymanjoined
14:35:57  * thlorenzjoined
14:39:24  * insertcoffeejoined
14:40:14  * thlorenzquit (Ping timeout: 240 seconds)
14:50:28  * thlorenzjoined
14:50:34  * timoxleyjoined
14:51:03  * timoxleyquit (Remote host closed the connection)
14:51:20  * timoxleyjoined
15:02:39  * jcrugzzjoined
15:38:06  * insertcoffeequit (Read error: Connection reset by peer)
15:40:28  * jcrugzzquit (Ping timeout: 240 seconds)
15:45:13  * esundahljoined
16:11:38  * Guest36601quit (Remote host closed the connection)
16:12:58  * dkjoined
16:37:52  * timoxleyquit (Remote host closed the connection)
16:38:28  * timoxleyjoined
16:42:41  * timoxleyquit (Ping timeout: 245 seconds)
16:47:20  * jcrugzzjoined
16:52:16  * jcrugzzquit (Ping timeout: 245 seconds)
17:09:35  * dominictarrquit (Quit: dominictarr)
17:09:52  * ncthom91joined
17:20:19  * thlorenzquit (Remote host closed the connection)
17:27:00  * jmartinsjoined
17:50:02  * insertcoffeejoined
17:51:27  * thlorenzjoined
18:00:18  * thlorenzquit (Ping timeout: 272 seconds)
18:04:41  * ncthom91quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
18:04:53  * mikealquit (Quit: Leaving.)
18:08:02  * mikealjoined
18:10:13  * jmartinsquit (Quit: Konversation terminated!)
18:26:41  * mikealquit (Quit: Leaving.)
18:35:16  * thlorenzjoined
18:41:22  * mikealjoined
18:43:24  * thlorenzquit (Remote host closed the connection)
18:57:47  * esundahlquit (Remote host closed the connection)
19:14:26  * esundahljoined
19:16:19  * mikealquit (Quit: Leaving.)
19:24:56  * insertcoffeequit (Remote host closed the connection)
19:42:20  * thlorenzjoined
19:42:29  * thlorenzquit (Remote host closed the connection)
19:46:57  * mikealjoined
20:11:00  * dominictarrjoined
20:22:57  * ncthom91joined
20:40:51  * mikealquit (Quit: Leaving.)
20:43:37  * thlorenzjoined
20:43:39  * dominictarrquit (Quit: dominictarr)
20:48:45  * thlorenzquit (Ping timeout: 272 seconds)
20:49:50  * mikealjoined
21:08:30  * ralphtheninjajoined
21:11:05  * ncthom91quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
21:27:51  * thlorenzjoined
21:36:52  * insertcoffeejoined
21:42:30  * jcrugzzjoined
21:48:15  * esundahlquit (Remote host closed the connection)
21:56:39  <levelbot>[npm] [email protected] <http://npm.im/wiki>: A Federated Wiki Server (@ward, @nrn)
22:14:13  * ncthom91_joined
22:17:15  <rescrv>rvagg: mbalho is not around, but I looked into that bug the two of you were talking about the other day. I can cause a crash under Node. I cannot crash when I push a similar workload from a C++ binding. I suspect that means that either there's a difference in the data loaded (I didn't get it perfect, just emulated sizes), or there's something about Node that corrupts the heap.
22:32:34  * jcrugzzquit (Ping timeout: 246 seconds)
22:40:27  * ncthom91_quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
23:09:37  * thlorenzquit (Remote host closed the connection)
23:22:08  <levelbot>[npm] [email protected] <http://npm.im/level-indico>: Simple indexing and querying for leveldb (@mariocasciaro)
23:27:18  * DTrejojoined
23:27:26  * DTrejoquit (Remote host closed the connection)
23:27:36  * DTrejojoined
23:34:59  * jcrugzzjoined
23:40:46  * thlorenzjoined
23:49:15  * thlorenzquit (Ping timeout: 272 seconds)
23:53:41  <levelbot>[npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg)
23:57:51  * jcrugzzquit (Ping timeout: 272 seconds)