00:09:42  <mbalho>rvagg: ping, need a sanity check on something
00:38:22  * mcollinajoined
00:42:46  * mcollinaquit (Ping timeout: 256 seconds)
01:09:09  * thlorenzjoined
01:19:52  * alanhoffquit
02:03:50  * werlequit (Ping timeout: 240 seconds)
02:15:38  * jcrugzzjoined
02:16:08  <mbalho>imported a 7 million row csv in 15 minutes... not bad
02:43:49  * ednapiranhajoined
02:44:06  * ednapiranhaquit (Remote host closed the connection)
02:54:08  * werlejoined
03:08:18  * timoxleyjoined
03:36:30  * timoxleyquit (Remote host closed the connection)
03:43:12  * timoxleyjoined
03:48:53  * levelbotquit (Remote host closed the connection)
03:49:08  * levelbotjoined
03:49:13  * levelbotquit (Remote host closed the connection)
03:49:28  * levelbotjoined
03:55:13  * esundahlquit (Remote host closed the connection)
03:55:46  * esundahljoined
04:00:10  * esundahlquit (Ping timeout: 256 seconds)
04:00:24  <thlorenz>were there some recent changes to sublevels streaming of keys?
04:00:45  <thlorenz>I'm pretty sure it didn't include the prefix in the keys it streamed, but now it does
04:01:04  <thlorenz>^ juliangruber mbalho
04:04:54  <thlorenz>rvagg: maybe there were changes in levelup regarding this?
04:05:17  <thlorenz>am getting keys like: 'ÿnpm-usersÿandrew' were I'm pretty sure I used to get just andrew
04:05:31  <thlorenz>npm-users is the sublevel prefix
04:18:47  <thlorenz>made an issue: https://github.com/dominictarr/level-sublevel/issues/30
04:26:22  * esundahljoined
04:33:21  * thlorenzquit (Remote host closed the connection)
04:34:18  * werlequit (Ping timeout: 240 seconds)
04:34:38  * esundahlquit (Ping timeout: 240 seconds)
04:47:53  <levelbot>[npm] [email protected] <http://npm.im/consensus>: vote for topics for your next meeting (@ceejbot)
04:49:27  <rvagg>mbalho: pong, sanity?
04:52:40  * werlejoined
05:56:38  * werlequit (Ping timeout: 240 seconds)
06:23:48  * werlejoined
06:32:04  * werlequit (Ping timeout: 246 seconds)
06:33:23  * mikealquit (Quit: Leaving.)
06:37:51  * mikealjoined
06:56:45  <mbalho>rvagg: figured it out! didnt realize that require('through') buffered when you pause
07:00:18  <mbalho>rvagg: i ended up with https://github.com/maxogden/level-bulk-load/issues/1#issuecomment-23617807 which seems to be working well
07:06:55  * DTrejojoined
07:09:16  * dominictarrjoined
07:31:46  <rvagg>mbalho: 'through2' is streams2 compliant btw, does proper backpressure and all that
07:43:42  * DTrejoquit (Remote host closed the connection)
07:47:49  * timoxleyquit (Remote host closed the connection)
07:50:27  * timoxley_joined
08:10:51  * timoxleyjoined
08:12:38  * timoxley_quit (Ping timeout: 245 seconds)
08:28:14  * rudquit (Quit: rud)
08:30:34  * werlejoined
08:35:18  * jcrugzzquit (Ping timeout: 264 seconds)
08:36:41  * werlequit (Ping timeout: 245 seconds)
08:42:41  * kenansulaymanjoined
08:49:36  * kenansulaymanquit (Quit: ≈ and thus my mac took a subtle yet profound nap ≈)
09:32:30  * Acconutjoined
09:40:00  * mcollinajoined
09:41:23  * jcrugzzjoined
09:46:16  * jcrugzzquit (Ping timeout: 245 seconds)
09:47:58  * timoxley_joined
09:48:14  * mcollinaquit (Ping timeout: 240 seconds)
09:50:39  * timoxleyquit (Ping timeout: 260 seconds)
10:02:40  * Acconutquit (Quit: Acconut)
10:02:46  * esundahljoined
10:03:47  * mcollinajoined
10:04:47  * timoxleyjoined
10:07:13  * timoxley_quit (Ping timeout: 245 seconds)
10:10:51  * esundahlquit (Ping timeout: 245 seconds)
10:15:23  * mcollinaquit (Ping timeout: 260 seconds)
11:22:19  * Acconutjoined
11:22:53  * rudjoined
11:22:53  * rudquit (Changing host)
11:22:53  * rudjoined
11:24:06  * rudquit (Client Quit)
11:25:34  * Acconutquit (Client Quit)
11:34:04  * Acconutjoined
11:34:12  * Acconutquit (Remote host closed the connection)
11:34:23  * Acconutjoined
11:34:24  * Acconutquit (Client Quit)
11:38:41  * Acconutjoined
11:43:10  * Acconutquit (Client Quit)
11:45:59  * Acconutjoined
11:47:02  * kenansulaymanjoined
11:50:41  * Acconutquit (Client Quit)
12:14:58  * Acconutjoined
12:16:24  * Acconutquit (Client Quit)
12:17:49  * rudjoined
12:17:49  * rudquit (Changing host)
12:17:49  * rudjoined
12:23:59  * fb55joined
12:27:22  <levelbot>[npm] [email protected] <http://npm.im/lmdb>: A Low-level, LevelDOWN-compatible, Node.js LMDB binding (@rvagg)
12:27:52  <kenansulayman>rvagg LMDB?
12:29:01  <kenansulayman>rvagg Symas MDB?
12:29:05  <rvagg>
12:30:09  <rvagg>started it months ago, just cleaned it up and made it release-worthy
12:30:12  <rvagg>.. sort of
12:30:16  <kenansulayman>rvagg Woah. Level vs MDB <=> 4.5mio ops/s vs 14.7 ops/s
12:30:52  <kenansulayman>Isn't there something like a limit for speed somewhen?
12:33:08  <rvagg>mmm? what do you mean?
12:33:47  <kenansulayman>Well I see new embedded databases outperform each other one after another
12:34:15  <rvagg>ah, right
12:34:18  <rescrv>kenansulayman: there's more too it with LevelDB vs LMDB
12:34:27  <rvagg>well, mdb comes with tradeoffs, even though Howard Chu claims there aren't
12:34:28  <rescrv>LMDB is good for large objects
12:34:31  <rvagg>it's a very complex story
12:34:35  <rescrv>LevelDB is better for small objects
12:34:50  <rescrv>LevelDB does great for scanning when you need the data sorted
12:34:57  <rescrv>LMDB won't necessarily do as well
12:35:00  <rvagg>I aint going to be switching level* to be using lmdb as a default backend any time soon
12:35:46  <rescrv>we have a port of HyperDex to LMDB, and it's faster for some workloads, and slower for others
12:37:22  <kenansulayman>Would a difference be visible at some point? I mean As a backend of a web-app 4.5 vs 14.7 isn't really something a user will notice
12:37:51  <rvagg>rescrv: and have you watched Howard crow about how LMDB beats the pants off LevelDB
12:37:55  <rvagg>he's quite the troll
12:39:01  <rvagg>kenansulayman: mostly these sorts of speed comparisons matter to very high workload services, which is why I keep on ranting on about how microbenchmarks are largely irrelevant for most developers because we just don't have the workload
12:39:32  <rvagg>it's starting to matter for mbalho because he's needing to push very large volumes around, but then he's only interested in very particular metrics of speed for his use-case
12:40:55  <kenansulayman>rvagg Still we're not even close to C10M… I'd go for the fastest then. Still there's too much overhead for any end-user to notice a change from 150ns to 75ns or so ;)
12:42:35  <kenansulayman>Anyway.. why does mbalho push that much data around?
12:44:42  <kenansulayman>brb
12:46:46  <rvagg>kenansulayman: cause of 'dat', you should check it out
12:46:58  <rvagg>an open-data platform, git for data, mainly academic data
12:47:04  <rvagg>of which there are large quantities
12:55:57  <rescrv>rvagg: there's always value in being more efficient, but Howard's comments are often read as "LSM-trees are inefficient", not "LevelDB has some inefficiencies compared to LMDB"
12:56:11  <rescrv>and he backs up his claims with benchmarks run on tmpfs, so everything is in memory
12:56:28  <rescrv>when pushed to do a workload that's 10x the size of memory he says no one would realistically do that
12:57:25  <rvagg>https://twitter.com/hyc_symas/status/369224875619020801
12:57:29  <rvagg>https://twitter.com/hyc_symas/status/369224944665624577
12:57:33  <rvagg>he's a troll
12:57:53  <rvagg>LevelDB is actually a very neat piece of engineering, as is LMDB
13:05:55  <rescrv>back when the whole LevelDB is for cellphones bit was going on, we asked Jeff Dean directly. He said it was meant for chrome and that they didn't want to rule out other ports, but just generally wanted a standalone kvs that used many of the techniques of big table
13:06:23  <rvagg>mm, which all makes sense
13:07:03  <rescrv>there's plenty of hints throughout the codebase that it could do better with a little more care
13:09:30  * fb55quit (Remote host closed the connection)
13:10:17  * Acconutjoined
13:11:21  * Acconutquit (Client Quit)
13:11:41  * Acconutjoined
13:15:25  <levelbot>[npm] [email protected] <http://npm.im/leveldown>: A Node.js LevelDB binding, primary backend for LevelUP (@rvagg)
13:15:52  <rvagg>^^ a minor release containing a few other (very minor) leaks I discovered but forgot to commit & publish
13:16:39  * Acconutquit (Quit: Acconut)
13:25:11  <kenansulayman>rvagg dat seems awesome. @ leveldown 0.8.1: changes?
13:27:09  <rvagg>kenansulayman: just some minor things, mainly missing HandleScope declarations; when you do that it seems to make V8 hold on to objects
13:28:15  <kenansulayman>I see
13:28:29  <kenansulayman>rvagg Could you merge them to leveldown-hyper? :D
13:29:02  <kenansulayman>Otherwise give me a sec and I'll do a PR
13:32:43  <kenansulayman>rvagg
13:50:47  <rvagg>geh, yeah sorry of course
13:52:53  <levelbot>[npm] [email protected] <http://npm.im/leveldown-hyper>: A Node.js LevelDB binding, primary backend for LevelUP (HyperDex fork) (@rvagg)
13:53:06  <kenansulayman>rvagg I was just to PR :(
13:53:23  <kenansulayman>thx anyway :D
13:54:52  <levelbot>[npm] [email protected] <http://npm.im/hyperlevel>: A Hyper-LevelDB wrapper (a convenience package bundling LevelUP & LevelDOWN-hyper) (@kenansulayman)
14:08:00  * Acconutjoined
14:09:35  * Acconutquit (Client Quit)
14:09:45  * fallsemojoined
14:14:10  * werlejoined
14:17:26  * fb55joined
14:20:22  * fallsemoquit (Quit: Leaving.)
14:20:29  <rescrv>rvagg: can you change 'HyperDex fork' to 'HyperLevelDB fork'?
14:23:32  * timoxley_joined
14:25:50  * timoxleyquit (Ping timeout: 240 seconds)
14:46:32  * thlorenzjoined
14:46:38  * rudquit (Quit: rud)
14:51:42  * esundahljoined
15:01:19  <thlorenz>rvagg: dominictarr: updated the sublevel keyStream issue: https://github.com/dominictarr/level-sublevel/issues/30
15:01:42  <thlorenz>it's clearly broken since my keys look different when I include values vs. when I don't
15:04:44  <dominictarr>thlorenz: what version of node are you on?
15:05:02  <thlorenz>v0.10.15
15:05:56  <thlorenz>dominictarr: I'm gonna upgrade to the latest 0.10 now and try again
15:06:58  <thlorenz>dominictarr: still seeing it with 0.10.17
15:08:56  <thlorenz>dominictarr: I'm also on MacOSX 10.7.5 (in case that matters)
15:12:43  <kenansulayman>juliangruber Known issues for level-store under 0.11?
15:13:53  <dominictarr>thlorenz: what levelup version?
15:15:12  <thlorenz>├─┬ [email protected]
15:15:12  <thlorenz>│ ├── [email protected]
15:15:12  <thlorenz>│ ├── [email protected]
15:15:13  <thlorenz>├─┬ [email protected]
15:15:15  <thlorenz>├─┬ [email protected]
15:15:17  <thlorenz>│ └── [email protected]
15:16:38  * mikealquit (Quit: Leaving.)
15:22:32  * fb55quit (Remote host closed the connection)
15:29:59  * Acconutjoined
15:29:59  * Acconutquit (Client Quit)
15:32:22  <dominictarr>thlorenz: you need to update sublevel
15:32:41  * mikealjoined
15:33:21  <thlorenz>dominictarr: I just did yesterday did anything change since then?
15:33:50  <thlorenz>ah I see 5 I had it pinned to 4.x
15:33:55  <thlorenz>will do
15:34:09  <dominictarr>yeah, now it supports the new streams2 read streams
15:36:18  <kenansulayman>dominictarr thlorenz You guys got 20 seconds to think of a word which is in row with super, hyper, ultra or mega
15:36:28  <dominictarr>uber
15:36:37  <thlorenz>duper
15:36:50  * missinglinkjoined
15:36:56  <dominictarr>ultimate
15:37:00  <thlorenz>dominictarr: still seeing the problem with newer sublevel
15:37:07  <dominictarr>hmm
15:37:17  <dominictarr>can you make a script to reproduce?
15:37:56  * mikealquit (Quit: Leaving.)
15:38:16  <thlorenz>I'll try give me a few mins
15:38:26  <dominictarr>thlorenz: I'm the level-sublevel tests pass, so that means we need a new test.
15:38:32  <thlorenz>yeah
15:44:50  <dominictarr>thlorenz: gonna go get food. back in a bit
15:45:06  <thlorenz>ok, I'll try to push up a repro case
15:45:10  <thlorenz>in the meantime
15:45:18  * missinglinkquit (Read error: Operation timed out)
15:50:56  * mcollinajoined
15:51:22  <levelbot>[npm] [email protected] <http://npm.im/hyperlevel>: A Hyper-LevelDB wrapper (a convenience package bundling LevelUP & LevelDOWN-hyper) (@kenansulayman)
15:52:12  * mcollina_joined
15:52:47  * fallsemojoined
15:52:52  <levelbot>[npm] [email protected] <http://npm.im/uberlevel>: A lmdb wrapper (a convenience package bundling LevelUP & lmdb) (@kenansulayman)
15:52:54  <kenansulayman>thlorenz dominictarr thx for the names^
15:54:02  * mcollinaquit (Read error: Connection reset by peer)
15:55:00  <kenansulayman>rvagg Uberlevel ftw
15:55:21  <rescrv>kenansulayman: uberlevel?
15:55:37  <kenansulayman>rescrv :D lmdb wrapper^
15:55:39  <rescrv>why not just llevel
15:55:57  <kenansulayman>I liked dominictarr's *uber*
15:57:00  * fallsemoquit (Ping timeout: 256 seconds)
15:57:30  * fb55joined
15:58:17  <rescrv>it's flashier
15:58:25  <rescrv>llevel just describes lmdb better
15:58:35  <rescrv>I think the "l" stands for lightning
15:58:54  <kenansulayman>yes
15:58:58  <kenansulayman>laserlevel :D
15:59:24  <kenansulayman>Reminds me of Robocop
15:59:29  <kenansulayman>how I loved that game
16:05:03  <thlorenz>dominictarr: pushed up an isolated repro: https://github.com/thlorenz/sublevel-keystream-bug
16:08:39  * mikealjoined
16:11:24  <levelbot>[npm] [email protected] <http://npm.im/valuepack-core>: Core utils and configurations for valuepack, not at all useful by itself. (@thlorenz)
16:13:19  <kenansulayman>thlorenz dominictarr That's because createReadStream doesn't return the emit.call => emit.call(this, 'data', val.substring(p.length))
16:13:52  <thlorenz>kenansulayman: so you tracked the bug already :) Nice!
16:14:17  <thlorenz>if it is legit we need to add a test to sublevel/ or levelup (whereever it belongs) and then fix it
16:14:24  <kenansulayman>Okay submitted a PR
16:14:36  <kenansulayman>https://github.com/KenanSulayman/level-sublevel/commit/d110d453f45b4950131d84c28b4bf30e35ac73ef
16:15:08  <kenansulayman>no wait
16:15:11  <thlorenz>kenansulayman: do you mind adding a failing test as well? you could use my repro as a basis
16:15:20  <kenansulayman>thlorenz Give me a sec
16:15:29  <thlorenz>sure take your time
16:16:34  <kenansulayman>{ key: 'dominictarr',
16:16:35  <kenansulayman> value: '{"contrib":"levelup"}' }
16:16:35  <kenansulayman>event !== data; event-type: readable
16:16:35  <kenansulayman>'ÿlevelup-usersÿdominictarr'
16:16:35  <kenansulayman>event === data; event-type: data
16:16:35  <kenansulayman>{ key: 'juliangruber',
16:16:35  <kenansulayman> value: '{"contrib":"multilevel"}' }
16:16:36  <kenansulayman>event !== data; event-type: readable
16:16:36  <kenansulayman>'ÿlevelup-usersÿjuliangruber'
16:16:37  <kenansulayman>event === data; event-type: data
16:16:37  <kenansulayman>{ key: 'rvagg',
16:16:38  <kenansulayman> value: '{"contrib":"leveldown"}' }
16:16:38  <kenansulayman>event !== data; event-type: readable
16:16:39  <kenansulayman>'ÿlevelup-usersÿrvagg'
16:16:58  <kenansulayman>seems to mix the data a bit
16:17:05  <thlorenz>yeah that's fine
16:17:25  <thlorenz>there are two streams running at the same time - you should go into index.js and comment one out at a time
16:17:30  <kenansulayman>no still looking into it wait
16:17:58  <thlorenz>this was just a very quick and dirty way to show the problem
16:19:27  <kenansulayman>That doesn't fix it
16:24:48  <thlorenz>kenansulayman: so you are seeing it too then - so first step is to add failing test - I can do that unless you want to do it
16:25:01  <kenansulayman>uhm your test is the failing test
16:25:17  <kenansulayman>http://data.sly.mn/image/2M1M281o2Y41
16:28:54  <kenansulayman>thlorenz It's an issue with tap
16:29:48  <kenansulayman>Because upusers.createReadStream({ keys: true, values: false }) .on('data', console.log) yiels the correct stuff
16:34:47  <thlorenz>kenansulayman: not an issue with tap, I assure you
16:35:23  <thlorenz>you are using ondata directly which works slightly different than piping
16:35:46  <thlorenz>kenansulayman: try: upusers.createReadStream({ keys: true, values: false }).pipe(through(console.log))
16:35:55  <thlorenz>you should get similar results
16:36:13  * Acconutjoined
16:36:34  <thlorenz>kenansulayman: still good finding though since it narrows the problem down a bit
16:37:57  * esundahlquit (Remote host closed the connection)
16:38:41  * mikealquit (Quit: Leaving.)
16:40:11  <kenansulayman>yup getting them
16:40:46  <kenansulayman>We can easily trace that by pipe through console.trace :)
16:44:37  * fb55quit (Remote host closed the connection)
16:48:52  <levelbot>[npm] [email protected] <http://npm.im/valuepack-core>: Core utils and configurations for valuepack, not at all useful by itself. (@thlorenz)
16:51:43  * rudjoined
16:55:53  <kenansulayman>thlorenz Ok here's what I think after node-inspecting it
16:57:02  <kenansulayman>the pipe receives the createReadStream data event of the root db (since sub.js : 176 tries to mutate the data)
16:58:23  <thlorenz>kenansulayman: any idea how to fix this?
16:58:31  <kenansulayman>since sub.js:169 returns an instance of root db's createReadStream
16:58:42  <kenansulayman>dominictarr Here?
16:58:54  <dominictarr>kenansulayman: houdy
16:59:01  <kenansulayman>traced down his issue
16:59:35  <kenansulayman>pretty much
16:59:40  <kenansulayman>Your word on it?
17:00:13  <thlorenz>kenansulayman: dominictarr: we should probably create a test case from the repro I pushed up first and then proceed to fix it ( https://github.com/thlorenz/sublevel-keystream-bug)
17:01:16  <thlorenz>kenansulayman: you up for adding that test case or should I do it?
17:01:23  <kenansulayman>just a sec
17:03:11  <dominictarr>thlorenz: does tap-stream check that two streams are equal?
17:03:20  <kenansulayman>Okay here's my finding:
17:03:28  <kenansulayman>two listeners on data
17:03:39  <thlorenz>dominictarr: tap-stream just pipes one stream through util.inspect
17:03:47  <kenansulayman>sublevels' listener doesn't mutate the data of the receiver
17:04:05  <thlorenz>the output is a bit garbled since index.js runs two stream in parallel - you should comment one out at a time to see better results
17:04:15  <kenansulayman>ÿlevelup-usersÿdominictarr
17:04:15  <kenansulayman>>>sublvl>> dominictarr
17:04:15  <kenansulayman>ÿlevelup-usersÿjuliangruber
17:04:15  <kenansulayman>>>sublvl>> juliangruber
17:04:15  <kenansulayman>ÿlevelup-usersÿrvagg
17:04:16  <kenansulayman>>>sublvl>> rvagg
17:04:34  <kenansulayman>>>sublvl>> is what the line looks inside sublevel before emitting it to the receiver
17:04:36  <dominictarr>thlorenz: so it's nothing to do with tape ?
17:04:43  <kenansulayman>the receiver get's a whole different data
17:05:11  <thlorenz>dominictarr: not at all it's tap-stream https://github.com/thlorenz/tap-stream
17:05:54  <dominictarr>I was confused because tape is named after tap (Test Anything Protocol)
17:06:17  * fb55joined
17:06:17  <thlorenz>yeah - my bad - name makes sense though since I'm tapping into the stream content ;)
17:06:31  <thlorenz>like a wire tap ;)
17:06:33  <dominictarr>inspect-stream?
17:07:04  <kenansulayman>lol I fixed it
17:07:05  <thlorenz>maybe - that module has been published a long time though, so I'd make some people unhappy to unpublish under the original name
17:07:25  <thlorenz>kenansulayman: awesome - can't wait for the PR
17:08:09  <dominictarr>you could leave the old one there… inspect-stream is available!
17:09:01  * esundahljoined
17:09:33  * Acconutquit (Quit: Acconut)
17:09:34  <dominictarr>thlorenz: but, do what you like
17:10:38  <thlorenz>dominictarr: sounds like an idea I guess I could add deprecate thing to tap-stream and unpublish in a year or so
17:11:03  <dominictarr>just never unpublish it
17:11:21  <dominictarr>and if you want to use the name later, bump major version.
17:12:26  <thlorenz>dominictarr: but what if someone else wants to use the name - I don't wanna hog names I don't use
17:13:15  <kenansulayman>dominictarr thlorenz My fix works. But the git version of sublevel works like a charm.. it's only the npm version which fails
17:13:51  <dominictarr>thlorenz: can you make it a test that fails, so I can just copy it into sublevel/test?
17:13:52  <levelbot>[npm] [email protected] <http://npm.im/lev>: commandline and REPL access for leveldb (@hij1nx)
17:13:55  <kenansulayman>Noticed that since the code was different from the npm patched version
17:14:06  <kenansulayman>dominictarr It doesn't fail on HEAD
17:14:17  <hij1nx>[ann] lev no shows sublevels in ls and autocomplete. Also, added a new feature `lev ./db --tree` will do the same as `npm ls` but for your sublevels. https://github.com/hij1nx/lev
17:14:31  <hij1nx>s/no/now/ ;)
17:14:32  <thlorenz>dominictarr: can do
17:14:35  <dominictarr>hij1nx: nice!
17:14:46  <thlorenz>hij1nx: cool! happy to use it now :)
17:14:51  <hij1nx>dominictarr: im working on nested autocompletes
17:15:03  <dominictarr>hij1nx: when do you get to ireland?
17:15:12  <hij1nx>dominictarr: the morning of the 7th
17:15:18  <hij1nx>^day before
17:15:30  <dominictarr>ah, sweet
17:17:11  <thlorenz>dominictarr: I'll just PR with a failing test is that ok?
17:17:30  * esundahlquit (Ping timeout: 264 seconds)
17:19:26  <kenansulayman>dominictarr Why does the npm version of sublevel differ from the git version?
17:19:48  <kenansulayman>repo is 5.0.1 and npm is 5.0.1
17:20:03  <kenansulayman>Though this is the npm version: http://data.sly.mn/text/0q3h3r0Y0H0s
17:20:28  <kenansulayman>And this the repo version: http://data.sly.mn/text/3K291v1s1q1y/contents
17:20:55  <kenansulayman>Wupps — npm: http://data.sly.mn/text/0q3h3r0Y0H0s/content and repo: http://data.sly.mn/text/3K291v1s1q1y/content
17:21:21  <thlorenz>kenansulayman: so the bug is not present in the git version?
17:21:42  <kenansulayman>thlorenz No. Even though they carry the same version they are completely different in code
17:21:51  <dominictarr>oops.
17:22:05  <dominictarr>something didn't publish properly!
17:22:10  <dominictarr>try now
17:22:20  <dominictarr>5.1.0
17:22:23  <levelbot>[npm] [email protected] <http://npm.im/level-sublevel>: partition levelup databases (@dominictarr)
17:22:40  * mikealjoined
17:22:42  <dominictarr>pushed to git also...
17:23:57  <kenansulayman>thlorenz:
17:24:08  <kenansulayman>http://data.sly.mn/text/0B0m0u1m3A3q
17:24:14  <kenansulayman>works on repo version
17:24:16  <kenansulayman>dominictarr ty
17:24:19  <thlorenz>kenansulayman: dominictarr: so my repro now doesn't repro anymore :)
17:24:29  <kenansulayman>;)
17:24:34  <thlorenz>dominictarr: do we still need the test?
17:25:12  <dominictarr>thlorenz: always
17:25:43  <thlorenz>it's gonna be hard to repro though, kenansulayman didn't you find it doesn't happen with on('data') at all?
17:26:18  <kenansulayman>thlorenz No the thing is my fix was to attach a listener using on("data") — the problem was fixed by that
17:26:23  <thlorenz>so I guess I'd have to collect the items in the test without calling on('data') -- any ideas dominictarr?
17:26:36  <thlorenz>all the current tests call on('data')
17:27:25  <kenansulayman>Gosh I need a coffee, enough bugs today
17:27:41  <thlorenz>kenansulayman: :)
17:28:36  <thlorenz>ah I guess I can use through - that doesn't call on('data')
17:28:47  * timoxley_quit (Remote host closed the connection)
17:29:05  <kenansulayman>thlorenz That's what caused the issue at the very beginning
17:29:32  <thlorenz>ok - well then I can use that fact to make sure the issue doesn't occur either way
17:29:38  <dominictarr>thlorenz: just pipe to through(array.push.bind(array), function () { console.log(array) })
17:29:52  <thlorenz>dominictarr: will do
17:30:06  <thlorenz>that's what I was thinking off ;)
17:30:58  <kenansulayman>dominictarr Can't you just do through(console.log) ?
17:30:59  * timoxleyjoined
17:34:43  <dominictarr>kenansulayman: need to collect them to make test
17:34:50  <kenansulayman>k
17:35:02  * timoxleyquit (Remote host closed the connection)
17:36:03  * Acconutjoined
17:40:00  * Acconutquit (Client Quit)
17:43:30  <thlorenz>dominictarr: https://github.com/dominictarr/level-sublevel/pull/32
17:43:45  <thlorenz>kenansulayman: I took care of the test
17:43:54  <kenansulayman>thlorenz ok great
17:45:19  * thlorenzquit (Remote host closed the connection)
17:45:48  * kenansulaymantopic: Too much backpressure
17:50:54  <levelbot>[npm] [email protected] <http://npm.im/lev>: commandline and REPL access for leveldb (@hij1nx)
17:50:54  <levelbot>[npm] [email protected] <http://npm.im/level-sublevel>: partition levelup databases (@dominictarr)
17:50:55  <dominictarr>kenansulayman: thlorenz thanks guys, merged
17:51:05  <kenansulayman>sure thing :)
17:52:18  <kenansulayman>http://c.thumbs.redditmedia.com/UNcO-h_QcS9PD-Gn.jpg
17:55:17  * mbalhotopic: Too much backpressure - logs: https://www.dropbox.com/s/7gz3779sxg6ygkw/%23%23leveldb.log
17:55:43  <kenansulayman>mbalho ------> http://logs.nodejs.org/leveldb/latest
17:56:23  <mbalho>ah well that should be in the topic then
17:56:31  * mbalhotopic: Too much backpressure - logs: http://logs.nodejs.org/leveldb/latest
17:56:32  <kenansulayman>wait
17:56:33  <kenansulayman>haha
17:56:34  <kenansulayman>ok
17:59:24  * rudquit (Quit: rud)
18:15:45  <hij1nx>A quick writeup about the new sublevel support in lev -- https://groups.google.com/forum/#!topic/node-levelup/mmGvx-8rbLw
18:18:54  <mbalho>hij1nx: whoa lev does a full table scan when it starts?
18:20:11  <mbalho>(in repl mode)
18:23:53  * fb55quit (Remote host closed the connection)
18:28:03  <mbalho>time lev test.db --keys | wc -l 7453008
18:28:04  <mbalho>real 4m36.497s
18:43:53  * esundahljoined
18:48:21  * esundahlquit (Ping timeout: 245 seconds)
18:48:23  * Acconutjoined
18:48:24  * Acconutquit (Client Quit)
18:55:18  * werlequit (Ping timeout: 264 seconds)
19:01:57  * i_m_cajoined
19:09:11  * thlorenzjoined
19:09:42  * sveisveiquit (Ping timeout: 264 seconds)
19:14:54  <levelbot>[npm] [email protected] <http://npm.im/valuepack-core>: Core utils and configurations for valuepack, not at all useful by itself. (@thlorenz)
19:18:40  * esundahl_joined
19:20:44  * esundahljoined
19:23:03  * esundahl_quit (Ping timeout: 245 seconds)
19:35:33  * rudjoined
19:35:33  * rudquit (Changing host)
19:35:33  * rudjoined
19:43:18  * reidquit (Ping timeout: 264 seconds)
19:45:11  * reidjoined
19:45:11  * reidquit (Changing host)
19:45:11  * reidjoined
19:47:26  * sveisveijoined
19:53:48  <dominictarr>rvagg: is gte lte available in leveldb yet?
20:15:42  * Acconutjoined
20:31:48  * i_m_caquit (Ping timeout: 245 seconds)
20:35:38  * Acconutquit (Quit: Acconut)
21:40:18  * mcollina_quit (Ping timeout: 264 seconds)
21:40:21  <hij1nx>mbalho: got any ideas for caching keys without a table scan?
21:40:40  <hij1nx>mbalho: i guess it could scan only each sublevel
21:42:17  * mcollinajoined
21:42:49  <hij1nx>mbalho: use case/objective criteria: if i have a database with that many keys, what kind of database would it be? would it be a database i would want to use lev with?
21:43:35  <mbalho>i can just use the cli to get single values but repls are nice sometimes
21:44:32  <hij1nx>mbalho: REPLS are nice for scripting with values, it would be cool if we could load really large ones too, im open to ideas for incrementally loading the cache
21:44:46  <mbalho>what is the cache for?
21:44:54  <hij1nx>mbalho: auto-complete
21:45:08  <mbalho>can you just have a flag that turns auto complete off
21:45:10  <hij1nx>mbalho: a lot of databases have long ugly keys, "getting" them can be a PITA
21:45:37  <hij1nx>mbalho: could do that, but autocomplete is a pretty major reason to use it in the first place
21:46:14  * mcollinaquit (Read error: Connection reset by peer)
21:46:52  * mcollinajoined
21:47:50  <hij1nx>mbalho: its super nice to type `get 6281nb<tab>` and then 32 more characters that would otherwise be unpleasnt to type is simply completed for you.
21:48:28  <mbalho>im not contesting that but it gets less useful as databases get larger
21:49:26  <mbalho>hij1nx: maybe you could do a lazy autocomplete that did a range when you hit tab
21:51:09  * mcollinaquit (Ping timeout: 240 seconds)
21:51:48  * mcollinajoined
21:54:32  <hij1nx>mbalho: hmm, ah! you gave me a great idea!
21:54:47  <hij1nx>mbalho: what about a zsh style-thing
21:55:04  <hij1nx>mbalho: where you lazy load, but page the results
21:55:49  <hij1nx>mbalho: so you'd start off only filling the page, if there is more to come, then you pause and break or resume
21:59:02  * jcrugzzjoined
22:19:22  <levelbot>[npm] [email protected] <http://npm.im/level-subtree>: build and maintain a tree from the sublevels in a leveldb instance (@hij1nx)
22:39:13  * i_m_cajoined
23:07:17  * dominictarrquit (Quit: dominictarr)
23:10:13  * kenansulaymanquit (Quit: ≈ and thus my mac took a subtle yet profound nap ≈)
23:11:28  * kenansulaymanjoined
23:19:13  * dominictarrjoined
23:27:52  <levelbot>[npm] [email protected] <http://npm.im/leveldown-basho>: A Node.js LevelDB (Basho fork) binding, primary backend for LevelUP (@rvagg)
23:28:07  <kenansulayman>rvagg haha
23:30:04  <rud>hmm anyone running levedown on a FreeBSD(9.x) host ? here npm install leveldown fails ..
23:30:17  <kenansulayman>rud What does it say?
23:30:20  <rud>(works flawlessly on my osx, same node build on both systems)
23:30:27  <rud>`sh "-c" "node-gyp rebuild"` failed with 1
23:30:39  <kenansulayman>Do you have msgpack in your app?
23:30:44  <rud>let me pastebin it
23:31:32  <rud>no, i do not, but can if it can provide useful infos
23:31:52  <rud>btw: http://pastebin.com/CwiY1czj
23:32:12  <kenansulayman>rud mom
23:32:22  <levelbot>[npm] [email protected] <http://npm.im/leveldown-basho>: A Node.js LevelDB (Basho fork) binding, primary backend for LevelUP (@rvagg)
23:33:27  <rud>hmm
23:33:40  <kenansulayman>rvagg Guess what
23:34:19  <rud>i have to admit i'm a bit confused regarding the available db adapters
23:34:22  <levelbot>[npm] [email protected] <http://npm.im/basholevel>: A Basho-LevelDB wrapper (a convenience package bundling LevelUP & Basho-LevelDB) (@kenansulayman)
23:34:25  * thlorenzquit (Remote host closed the connection)
23:34:26  <kenansulayman>:D :D
23:35:05  <rvagg>kenansulayman: it'd be ideal if you could keep the versions the same as the levelup versions, that's what we do with 'level'
23:35:29  <kenansulayman>rvagg Ah yes sorry. was still the lmdb verion
23:35:45  <rvagg>and you really need to be exporting more than just the function
23:36:11  <rvagg>https://github.com/rvagg/node-levelup#destroy
23:36:14  <rvagg>https://github.com/rvagg/node-levelup#repair
23:36:20  <dominictarr>rvagg: is this hard-coded assembly? https://github.com/rvagg/node-leveldown/blob/20c28de6087a4a71d8ac1813245e199e524e6b3d/deps/leveldb/leveldb-1.11.0/port/atomic_pointer.h#L199-L204
23:36:21  <levelbot>[npm] [email protected] <http://npm.im/basholevel>: A Basho-LevelDB wrapper (a convenience package bundling LevelUP & Basho-LevelDB) (@kenansulayman)
23:36:31  <rud>same error with basholevel, grrr
23:36:46  <rvagg>so, it'd be best if you iterate through anything that's hanging off the 'levelup' object and copy it to your exports
23:36:50  <kenansulayman>rud lol try hyperlevel ;)
23:36:56  <kenansulayman>rud Okay, mom
23:37:14  <kenansulayman>rud Then try uberlevel :D
23:37:34  <rescrv>rud: is it a compile error?
23:37:42  <rescrv>does hyperlevel work?
23:38:52  * dominictarrquit (Quit: dominictarr)
23:38:54  <rud>rescrv: no: http://pastebin.com/index/CwiY1czj
23:39:41  <kenansulayman>rud That's not the result of the hyperlevel installation
23:39:49  <rud>sorry, paste bin issue
23:41:04  <rud>here it is http://pastebin.com/mStyi5tZ
23:41:33  <rescrv>rvagg: for node-leveldown issue 58, I have private patches to HyperLevelDB that bring 30%+ speedup to the skip list. They need further testing though.
23:41:35  <rud>i think they all fail because of the OS version
23:41:40  <rud>something with os_include ?
23:41:54  <rud>(line 35)
23:41:54  <rvagg>rud: that pastebin is oddly sparse on actual details, could you give us /home/www/iconference/sources/socket_router/npm-debug.log perhaps?
23:42:13  <rvagg>rud: and also try `sudo npm install npm -g` to get the latest npm & node-gyp just in case
23:42:15  <rud>rvagg: sure, anything i can provide
23:42:29  * jcrugzzquit (Ping timeout: 246 seconds)
23:42:31  <rud>ok
23:45:11  * thlorenzjoined
23:46:19  <rud>i did update npm, and both dev (osx) and prod (freebsd) share the same npm build
23:46:34  <rud>i'm trying to figure where/why my npm-debug.log vanish
23:49:22  <levelbot>[npm] [email protected] <http://npm.im/hyperlevel>: A Hyper-LevelDB wrapper (a convenience package bundling LevelUP & LevelDOWN-hyper) (@kenansulayman)
23:49:22  <levelbot>[npm] [email protected] <http://npm.im/basholevel>: A Basho-LevelDB wrapper (a convenience package bundling LevelUP & Basho-LevelDB) (@kenansulayman)
23:49:51  <rud>current npm install hyperlevel: http://pastebin.com/KQix7JuB will post logs.. when i find them ;)
23:49:53  <levelbot>[npm] [email protected] <http://npm.im/uberlevel>: A lmdb wrapper (a convenience package bundling LevelUP & lmdb) (@kenansulayman)
23:49:58  <kenansulayman>rvagg There you go
23:50:15  <rvagg>uber? heh
23:51:15  <rvagg>rud: ahh.. "os_include"
23:52:21  <rvagg>rud: so, we have linux, mac, windows, solaris as "officially" supported
23:53:07  <rud>rvagg: i'm willing to do whatever my humble competence can provide to add freebsd to that list :/
23:53:19  <kenansulayman>haha
23:53:40  <kenansulayman>Still FreeBSD is unixoid right?
23:53:50  <rud>yes, if it runs on osx, it should on bsd
23:54:12  <kenansulayman>nah that's too far for an expectation^
23:54:32  <rud>also, this really catches my attention, line 35 http://pastebin.com/KQix7JuB can't it be that the OS name is just missing ?
23:55:10  <kenansulayman>Nah it just says the snappy lib can't be successfully compiled on your machine
23:56:05  <rvagg>rud: yeah, can you clone the leveldown repo, `sudo npm install node-gyp -g`, `npm install` in the leveldown repo and I'll talk you through some stuff to test with
23:56:29  <rud>sure, give me a few secs to set it up
23:56:58  <kenansulayman>http://data.sly.mn/text/3B322z0b3E3D/contents it's not listed
23:57:17  <kenansulayman>Can't we just try to see if it compiles with the std unix settings?
23:58:33  <kenansulayman>rvagg That'd be the identifier "openbsd" right?
23:59:53  <kenansulayman>sysv5* | sco3.2v5* | sco5v6* | unixware* | OpenUNIX* | sysv4*uw2*) <= bsd is listed for snappy