| 00:09:21
| * mikeal | joined |
| 00:10:27
| * wolfeidau | joined |
| 00:11:34
| * wolfeida_ | joined |
| 00:15:29
| * wolfeidau | quit (Ping timeout: 265 seconds) |
| 00:19:27
| * mikeal | quit (Ping timeout: 272 seconds) |
| 00:32:50
| * wolfeida_ | quit (Remote host closed the connection) |
| 00:35:22
| * wolfeida_ | joined |
| 00:44:12
| <brycebaril> | rvagg: is there any way to know if {sync: true} is working? In my batch load testing I can't really see a difference. |
| 00:46:41
| <brycebaril> | mostly I'm trying to figure out a way to signal backpressure to a writeStream wrapper. I think what I need now is some way to signal back-pressure upstream |
| 00:49:19
| <brycebaril> | If I avoid objectMode streams & key/value encoding logic with pure binary (multibuffer) streams, I can significantly cut the memory & time, but it appears to hit a point where I'm still ending up queing most/all of it into memory |
| 00:49:31
| * julianduque | quit (Ping timeout: 246 seconds) |
| 00:49:41
| <levelbot> | [npm] [email protected] <http://npm.im/level-assoc>: relational foreign key associations (hasMany, belongsTo) for leveldb (@substack) |
| 00:52:22
| <substack> | row.blah({ limit: n }) works yay |
| 00:52:39
| * thlorenz | joined |
| 00:53:04
| <brycebaril> | I'll try a manual throttle to see if that helps for now, but it would be nice if there was something exposed in the LevelDB api that correlated with write task readiness (or LOG->table pending work) |
| 00:56:51
| * thlorenz | quit (Ping timeout: 245 seconds) |
| 01:03:57
| * fb55 | joined |
| 01:04:22
| * fb55 | quit (Remote host closed the connection) |
| 01:04:48
| * fb55 | joined |
| 01:09:28
| * fb55 | quit (Ping timeout: 246 seconds) |
| 01:15:44
| * mikeal | joined |
| 01:18:12
| * wolfeida_ | quit (Remote host closed the connection) |
| 01:20:37
| * mikeal | quit (Ping timeout: 272 seconds) |
| 01:29:07
| <brycebaril> | rvagg: Hmm... well, duh -- for bulk loading {sync: true} wouldn't matter. Is there anyway to expose a sync batch that *does* behave synchronously WRT Node? |
| 01:30:28
| * zz_eugeneware | changed nick to eugeneware |
| 01:35:46
| <rvagg> | brycebaril: in terms of blocking behaviour you *could* expose a synchronous/blocking batch, but we're not going to do that; ideally the callback should be all you need but what I'm hearing is that it isn't for this bulk load situation |
| 01:37:58
| <rvagg> | it would be lovely to have an event system for leveldb that gave us insight into compactions and other internal behaviours |
| 01:38:19
| <rvagg> | not impossible but it would mean properly digging in to the guts of leveldb to expose it, which isn't very friendly for an upgrade path |
| 01:38:24
| <rvagg> | ... but it would be interesting! |
| 01:41:15
| * wolfeidau | joined |
| 01:41:45
| * esundahl | joined |
| 01:46:52
| * wolfeidau | quit (Remote host closed the connection) |
| 01:47:24
| * wolfeidau | joined |
| 01:49:15
| * rud | joined |
| 01:49:38
| * thlorenz | joined |
| 01:49:51
| <rvagg> | I'm thinking that we should perhaps make our own Log implementation that can be plugged in to and streamed so you can watch progress and/or pipe it to an actual log somewhere if you like |
| 01:52:17
| * wolfeidau | quit (Ping timeout: 272 seconds) |
| 02:04:06
| * rud | quit (Quit: rud) |
| 02:51:00
| <nathan7> | rvagg: does LevelDB have any dtraceness? |
| 02:53:40
| * thlorenz | quit (Remote host closed the connection) |
| 02:54:00
| <ogd> | rvagg: from what i can tell compaction is important for bulk loading stuff but theres no visibility into what the compaction state is |
| 02:54:23
| <ogd> | rvagg: the callback for a batch fires and compaction might still be happening |
| 02:55:27
| <ogd> | rvagg: but i think its only an issue if you are writing too much data |
| 02:56:38
| <ogd> | so i guess the answer is to not write too much data |
| 02:57:06
| <ogd> | i'm still interested to see someone test out that multiple leveldbs for writing that get combined later idea |
| 02:57:55
| * ramitos | joined |
| 03:06:01
| * nnnnathann | quit (Ping timeout: 245 seconds) |
| 03:12:31
| * ramitos | quit (Quit: Computer has gone to sleep.) |
| 03:14:16
| * ramitos | joined |
| 03:24:49
| * thlorenz | joined |
| 03:30:13
| * ramitos | quit (Quit: Textual IRC Client: www.textualapp.com) |
| 03:32:48
| * thlorenz | quit (Ping timeout: 240 seconds) |
| 03:38:03
| * jcrugzz | joined |
| 03:45:23
| * dominictarr | joined |
| 03:51:34
| * dominictarr | quit (Quit: dominictarr) |
| 03:52:08
| * timoxley | joined |
| 03:52:22
| <juliangruber> | substack: sweet |
| 04:18:49
| * jcrugzz | quit (Ping timeout: 246 seconds) |
| 04:29:43
| * thlorenz | joined |
| 04:34:25
| * thlorenz | quit (Ping timeout: 272 seconds) |
| 04:49:07
| <rvagg> | ogd: https://github.com/rvagg/node-leveldown/pull/67 |
| 04:59:11
| * jxson | quit (Remote host closed the connection) |
| 05:04:22
| <ogd> | rvagg: nice |
| 05:12:45
| <brycebaril> | rvagg: awesome, I'll take a look. Yeah, I don't think the bulk load situation needs much, just some way of propogating backpressure. The issue is with the async callbacks there is nothing to tell upstream to pause |
| 05:15:52
| <rvagg> | yeah.. using log data for anything useful would be pretty hacky; but the info is interesting and educational nonetheless |
| 05:45:13
| <brycebaril> | rvagg: what about deferring the batch callback until the sync was done if {sync: true}? That way the api doesn't change, just timing if they request synchronous writes? |
| 05:45:30
| <rvagg> | brycebaril: that's the way it should happen |
| 05:47:31
| * esundahl | quit (Remote host closed the connection) |
| 05:47:44
| <brycebaril> | Oh? Hmm, lemme see if I can see a timing difference between batches with sync true or false. Maybe it is something else. |
| 05:48:39
| <levelbot> | [npm] [email protected] <http://npm.im/memdown>: An drop-in replacement for LevelDOWN that works in memory only (@rvagg) |
| 05:50:02
| * mikeal | joined |
| 05:50:06
| <brycebaril> | I'm still trying to figure out what is taking up heapspace memory in my bulk loader. So if buffers mostly live outside heap, any idea what the heap memory component looks like for them? |
| 05:53:40
| <levelbot> | [npm] [email protected] <http://npm.im/memdown>: An drop-in replacement for LevelDOWN that works in memory only (@rvagg) |
| 05:59:09
| <rvagg> | join #libuv and bother trevnorris |
| 05:59:23
| <ogd> | hmm something im doing in dat is slowing reads down to 7k/s whereas i get ~100k/s with normal levelup... need to investigate |
| 05:59:29
| <rvagg> | actually, trevnorris is in here |
| 06:05:10
| <levelbot> | [npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg) |
| 06:07:39
| <levelbot> | [npm] [email protected] <http://npm.im/level-mem>: A convenience package bundling LevelUP & MemDOWN (@rvagg) |
| 06:09:11
| * eugeneware | changed nick to zz_eugeneware |
| 06:19:54
| <brycebaril> | Yeah, I get the same mean/median timing for batches if sync is true or false |
| 06:22:46
| <rvagg> | that's odd |
| 06:23:04
| <rvagg> | chained batch or array batch? |
| 06:23:20
| <brycebaril> | array |
| 06:24:30
| <rvagg> | I'll double-check that sync is getting properly passed down to leveldb when I have time later tonight, it's impossible to unit test for unfortunately |
| 06:24:56
| <rvagg> | can't mock out kernel fs writes unfortunately! |
| 06:28:07
| <brycebaril> | I'm still getting 28k records per second, which isn't bad by any means |
| 06:29:33
| <brycebaril> | Apparently memwatch heapdump on a 700+ meg heapspace isn't a good idea |
| 06:29:44
| <brycebaril> | 22449 bryce 20 0 11.6g 6.5g 788 R 26 84.2 5:29.16 node |
| 06:45:26
| <brycebaril> | ok, solved my memory problem |
| 06:45:41
| <brycebaril> | array batch = bad for memory |
| 06:47:43
| <brycebaril> | rvagg: memory use for standard levelup rs.pipe(ws) for a 5 million record database: (1994.csv that ogd uses): 1364895432 |
| 06:48:06
| <brycebaril> | using the buffers-only read/write streams I'm working on: 13701672 |
| 06:48:12
| <brycebaril> | those are JS heap space |
| 06:49:01
| <brycebaril> | rss 3_153_367_040 vs 554_786_816 |
| 06:50:24
| <brycebaril> | timing 257s -> 182s |
| 06:58:31
| <rvagg> | this is why writestream has to go, it's a bikeshed |
| 07:02:52
| <brycebaril> | I'm over worrying about keeping it in the core. I'll still use some sort of writeStream interface, but I know what I'm doing. |
| 07:03:38
| <brycebaril> | If someone with a big level instance takes the README's advice about making a copy() function they are guaranteed to have a bad experience. That's probably reason enough to pull it. |
| 07:06:34
| <rvagg> | in fact there's a .copy() exported on LevelUP that probably shouldn't be there |
| 07:06:57
| <rvagg> | see latest comment on #199 |
| 07:07:29
| <brycebaril> | I'm assuming by bikeshed you mean that any given developer/application is going to see a different need for writeStream that needs a completely different implementation? If so, I completely agree. |
| 07:15:09
| * wolfeidau | joined |
| 07:16:03
| * zz_eugeneware | changed nick to eugeneware |
| 07:33:15
| * eugeneware | changed nick to zz_eugeneware |
| 08:46:03
| * fb55 | joined |
| 08:46:07
| * dominictarr | joined |
| 09:07:41
| <levelbot> | [npm] [email protected] <http://npm.im/level-ws>: A basic WriteStream implementation for LevelUP (@rvagg) |
| 09:07:58
| * fb55 | quit (Remote host closed the connection) |
| 09:08:25
| * fb55 | joined |
| 09:12:49
| * fb55 | quit (Ping timeout: 246 seconds) |
| 09:17:40
| <levelbot> | [npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg) |
| 09:32:53
| * thlorenz | joined |
| 09:33:41
| <levelbot> | [npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg) |
| 09:35:56
| * pgte | joined |
| 09:37:26
| * thlorenz | quit (Ping timeout: 264 seconds) |
| 09:38:47
| * timoxley | quit (Remote host closed the connection) |
| 09:44:46
| <rvagg> | FYI anyone not following #199 (remove WriteStream) https://github.com/rvagg/node-levelup/issues/199#issuecomment-26194464 |
| 09:51:03
| * fb55 | joined |
| 09:56:39
| * pgte | quit (Remote host closed the connection) |
| 09:57:11
| * pgte | joined |
| 10:01:47
| * pgte | quit (Ping timeout: 265 seconds) |
| 10:16:03
| * zz_eugeneware | changed nick to eugeneware |
| 10:17:02
| * fb55 | quit (Ping timeout: 240 seconds) |
| 10:24:53
| * fb55 | joined |
| 11:55:23
| * dominictarr | quit (Quit: dominictarr) |
| 11:56:52
| * dominictarr | joined |
| 11:58:18
| * dominictarr | quit (Client Quit) |
| 11:59:28
| * eugeneware | changed nick to zz_eugeneware |
| 12:19:46
| * thlorenz | joined |
| 12:22:09
| * jez0990 | quit (Remote host closed the connection) |
| 12:24:57
| * jez0990 | joined |
| 12:25:37
| * thlorenz | quit (Ping timeout: 272 seconds) |
| 12:28:54
| * jez0990 | quit (Read error: Connection reset by peer) |
| 12:28:55
| * dominictarr | joined |
| 12:31:53
| * carolyns | changed nick to foxcat |
| 12:31:53
| <dominictarr> | rvagg: hey, did you document your setup for your nexus7/rpi thing? |
| 12:32:13
| * foxcat | part |
| 12:35:19
| <hij1nx> | i noticed that goatdb has migrated to australia... |
| 12:40:09
| <rvagg> | dominictarr: no, no documentation, typical linux hackety hack |
| 12:40:29
| <rvagg> | dominictarr: if you need it I guess I can |
| 12:41:00
| <rvagg> | hij1nx: where is it? |
| 12:41:08
| <dominictarr> | rvagg: main question: did you use a seprate router? how do you get the rpi to connect to the right network? |
| 12:41:26
| <rvagg> | dominictarr: linux wifi tools |
| 12:41:36
| <rvagg> | dominictarr: pi has a wifi dongle and a bluetooth dongle |
| 12:41:57
| <rvagg> | the bluetooth is the most important, it's set up so that I can connect to it from the nexus |
| 12:42:04
| <dominictarr> | so, you connect to the nexus7 via bt? |
| 12:42:08
| <rvagg> | I can pull out the wifi and it still works, I just get no external networking |
| 12:42:15
| <rvagg> | yeah, bluetooth from nexus to pi |
| 12:42:21
| <rvagg> | bluetooth networking to be exact |
| 12:42:27
| <dominictarr> | right - I see |
| 12:42:32
| <rvagg> | that was a bit of a pain but I think I could probably document the recipe that I landed on |
| 12:43:13
| <rvagg> | it's always predictable, the pi shows up with ip address 192.168.200.1 which I start a session on when the bluetooth is configured |
| 12:43:32
| <rvagg> | then when I'm in I manually configure whatever wifi network is available using wpa_supplicant ... grr ... linux wifi crap |
| 12:43:49
| * rud | joined |
| 12:43:49
| * rud | quit (Changing host) |
| 12:43:49
| * rud | joined |
| 12:44:24
| <rvagg> | ingredients: pi, bluetooth dongle pi compatible, wifi dongle pi compatible, nexus 7, bluetooth keyboard for nexus 7, battery pack for pi and nexus and keyboard where required |
| 12:45:43
| <rvagg> | the last leg of my trip home I had the pi & battery tucked away in my bag in the overhead baggage locker and it never came out, I just fired up the nexus when I was ready to hack and connected via bluetooth |
| 12:46:25
| <rvagg> | I do have a problem with repeat bluetooth connections, it doesn't seem to reset well so as insurance if I'm disconnecting the nexus from the pi and I know I'll want to reconnect later I'll tell it to reboot just to start fresh and so I don't have to rip the power out and put it back in again to force a restart |
| 12:46:36
| <rvagg> | but that's something that just needs tweaking so I can reconnect later |
| 12:46:47
| <rvagg> | otherwise it's pretty stable and works well |
| 12:46:59
| <rvagg> | and I actually got a fair bit of work done on the plane this time, more than I've done before |
| 12:49:03
| <rvagg> | heh, pomke has goatdb now |
| 12:49:47
| * rvagg | bed |
| 12:56:19
| * zz_eugeneware | changed nick to eugeneware |
| 13:23:53
| * fb55 | quit (Remote host closed the connection) |
| 13:24:20
| * fb55 | joined |
| 13:26:41
| * rud_ | joined |
| 13:28:26
| * rud | quit (Ping timeout: 264 seconds) |
| 13:28:27
| * rud_ | changed nick to rud |
| 13:29:35
| * fb55 | quit (Ping timeout: 272 seconds) |
| 13:34:05
| * dominictarr | quit (Quit: dominictarr) |
| 13:58:41
| * pgte | joined |
| 14:04:51
| * dominictarr | joined |
| 14:17:09
| * eugeneware | changed nick to zz_eugeneware |
| 14:18:46
| * zz_eugeneware | changed nick to eugeneware |
| 14:19:11
| * pgte | quit (Remote host closed the connection) |
| 14:24:30
| * pgte | joined |
| 14:25:17
| * pgte | quit (Remote host closed the connection) |
| 14:29:33
| * kenansulayman | joined |
| 14:35:57
| * thlorenz | joined |
| 14:39:24
| * insertcoffee | joined |
| 14:40:14
| * thlorenz | quit (Ping timeout: 240 seconds) |
| 14:50:28
| * thlorenz | joined |
| 14:50:34
| * timoxley | joined |
| 14:51:03
| * timoxley | quit (Remote host closed the connection) |
| 14:51:20
| * timoxley | joined |
| 15:02:39
| * jcrugzz | joined |
| 15:38:06
| * insertcoffee | quit (Read error: Connection reset by peer) |
| 15:40:28
| * jcrugzz | quit (Ping timeout: 240 seconds) |
| 15:45:13
| * esundahl | joined |
| 16:11:38
| * Guest36601 | quit (Remote host closed the connection) |
| 16:12:58
| * dk | joined |
| 16:37:52
| * timoxley | quit (Remote host closed the connection) |
| 16:38:28
| * timoxley | joined |
| 16:42:41
| * timoxley | quit (Ping timeout: 245 seconds) |
| 16:47:20
| * jcrugzz | joined |
| 16:52:16
| * jcrugzz | quit (Ping timeout: 245 seconds) |
| 17:09:35
| * dominictarr | quit (Quit: dominictarr) |
| 17:09:52
| * ncthom91 | joined |
| 17:20:19
| * thlorenz | quit (Remote host closed the connection) |
| 17:27:00
| * jmartins | joined |
| 17:50:02
| * insertcoffee | joined |
| 17:51:27
| * thlorenz | joined |
| 18:00:18
| * thlorenz | quit (Ping timeout: 272 seconds) |
| 18:04:41
| * ncthom91 | quit (Quit: My MacBook has gone to sleep. ZZZzzz…) |
| 18:04:53
| * mikeal | quit (Quit: Leaving.) |
| 18:08:02
| * mikeal | joined |
| 18:10:13
| * jmartins | quit (Quit: Konversation terminated!) |
| 18:26:41
| * mikeal | quit (Quit: Leaving.) |
| 18:35:16
| * thlorenz | joined |
| 18:41:22
| * mikeal | joined |
| 18:43:24
| * thlorenz | quit (Remote host closed the connection) |
| 18:57:47
| * esundahl | quit (Remote host closed the connection) |
| 19:14:26
| * esundahl | joined |
| 19:16:19
| * mikeal | quit (Quit: Leaving.) |
| 19:24:56
| * insertcoffee | quit (Remote host closed the connection) |
| 19:42:20
| * thlorenz | joined |
| 19:42:29
| * thlorenz | quit (Remote host closed the connection) |
| 19:46:57
| * mikeal | joined |
| 20:11:00
| * dominictarr | joined |
| 20:22:57
| * ncthom91 | joined |
| 20:40:51
| * mikeal | quit (Quit: Leaving.) |
| 20:43:37
| * thlorenz | joined |
| 20:43:39
| * dominictarr | quit (Quit: dominictarr) |
| 20:48:45
| * thlorenz | quit (Ping timeout: 272 seconds) |
| 20:49:50
| * mikeal | joined |
| 21:08:30
| * ralphtheninja | joined |
| 21:11:05
| * ncthom91 | quit (Quit: My MacBook has gone to sleep. ZZZzzz…) |
| 21:27:51
| * thlorenz | joined |
| 21:36:52
| * insertcoffee | joined |
| 21:42:30
| * jcrugzz | joined |
| 21:48:15
| * esundahl | quit (Remote host closed the connection) |
| 21:56:39
| <levelbot> | [npm] [email protected] <http://npm.im/wiki>: A Federated Wiki Server (@ward, @nrn) |
| 22:14:13
| * ncthom91_ | joined |
| 22:17:15
| <rescrv> | rvagg: mbalho is not around, but I looked into that bug the two of you were talking about the other day. I can cause a crash under Node. I cannot crash when I push a similar workload from a C++ binding. I suspect that means that either there's a difference in the data loaded (I didn't get it perfect, just emulated sizes), or there's something about Node that corrupts the heap. |
| 22:32:34
| * jcrugzz | quit (Ping timeout: 246 seconds) |
| 22:40:27
| * ncthom91_ | quit (Quit: My MacBook has gone to sleep. ZZZzzz…) |
| 23:09:37
| * thlorenz | quit (Remote host closed the connection) |
| 23:22:08
| <levelbot> | [npm] [email protected] <http://npm.im/level-indico>: Simple indexing and querying for leveldb (@mariocasciaro) |
| 23:27:18
| * DTrejo | joined |
| 23:27:26
| * DTrejo | quit (Remote host closed the connection) |
| 23:27:36
| * DTrejo | joined |
| 23:34:59
| * jcrugzz | joined |
| 23:40:46
| * thlorenz | joined |
| 23:49:15
| * thlorenz | quit (Ping timeout: 272 seconds) |
| 23:53:41
| <levelbot> | [npm] [email protected] <http://npm.im/level-packager>: LevelUP package helper for distributing with a LevelDOWN-compatible back-end (@rvagg) |
| 23:57:51
| * jcrugzz | quit (Ping timeout: 272 seconds) |