00:22:34  * tmcwjoined
00:28:27  * esundahl_quit (Remote host closed the connection)
00:28:56  * esundahljoined
00:29:26  * tmcwquit
00:29:43  * DTrejojoined
00:33:47  * esundahlquit (Ping timeout: 272 seconds)
00:36:49  * jerrysvquit (Quit: Leaving...)
00:42:20  * DTrejoquit (Remote host closed the connection)
01:12:15  * funkytekquit (Ping timeout: 246 seconds)
01:14:38  * jjmalinaquit (Ping timeout: 264 seconds)
01:27:47  * stagasjoined
01:31:13  * Aredrideljoined
01:36:37  * ramitosquit (Ping timeout: 246 seconds)
01:43:35  * daviddiasjoined
01:54:17  * kenansulaymanchanged nick to kenan|afk
01:56:22  * thlorenzjoined
02:04:45  * daviddiasquit (Read error: Connection reset by peer)
02:07:03  * daviddiasjoined
02:13:30  * scttnlsnjoined
02:19:40  * jjmalinajoined
02:19:45  * jjmalinaquit (Client Quit)
03:00:53  * scttnlsnquit (Remote host closed the connection)
03:03:04  * DTrejojoined
03:03:25  * TehShrikequit (Quit: Leaving.)
03:08:38  * TehShrikejoined
03:15:36  * DTrejoquit
03:21:53  * TehShrikequit (Quit: Leaving.)
03:25:32  * TehShrikejoined
03:27:55  * tmcwjoined
03:28:52  * tmcwquit (Client Quit)
04:02:25  * jxsonjoined
04:02:35  * paulfryzelquit (Remote host closed the connection)
04:03:49  * jxsonquit (Remote host closed the connection)
04:08:25  * daviddiasquit (Read error: Connection reset by peer)
04:14:48  * funkytekjoined
04:25:34  * jxsonjoined
04:39:18  * timoxleyquit (Remote host closed the connection)
04:51:53  * timoxleyjoined
04:53:32  * timoxleyquit (Remote host closed the connection)
05:01:55  * scttnlsnjoined
05:05:07  * timoxleyjoined
05:06:14  * timoxleyquit (Read error: Connection reset by peer)
05:06:38  * timoxleyjoined
05:06:44  * scttnlsnquit (Ping timeout: 265 seconds)
05:06:51  * timoxleyquit (Remote host closed the connection)
05:07:26  * thlorenzquit (Remote host closed the connection)
05:27:52  * jxsonquit (Remote host closed the connection)
05:36:18  * jcrugzzjoined
05:42:50  * Aredridelquit (Quit: Textual IRC Client: www.textualapp.com)
05:49:31  * esundahljoined
05:58:26  * levelbotjoined
06:10:49  * levelbotquit (Remote host closed the connection)
06:11:08  * levelbotjoined
06:12:15  * jcrugzzquit (Ping timeout: 240 seconds)
06:12:53  * funkytekquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
06:13:02  * jcrugzzjoined
06:25:02  * funkytekjoined
06:47:15  * mikealquit (Quit: Leaving.)
06:48:18  * mikealjoined
06:52:49  * mikealquit (Ping timeout: 252 seconds)
06:56:14  * mikealjoined
07:02:57  * scttnlsnjoined
07:03:17  * scttnlsnquit (Read error: Connection reset by peer)
07:15:44  * esundahlquit (Remote host closed the connection)
07:17:43  * levelbotquit (Remote host closed the connection)
07:18:19  * levelbotjoined
07:28:41  <levelbot>[npm] [email protected] <http://npm.im/dynalite>: A mock implementation of Amazon's DynamoDB built on LevelDB (@hichaelmart)
07:36:12  <levelbot>[npm] [email protected] <http://npm.im/dynalite>: A mock implementation of Amazon's DynamoDB built on LevelDB (@hichaelmart)
08:20:25  * funkytekquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
08:33:45  <levelbot>[npm] [email protected] <http://npm.im/model>: Datastore-agnostic ORM in JavaScript (@mde)
08:42:05  * contraha_changed nick to _Contra
09:03:44  * scttnlsnjoined
09:07:17  * funkytekjoined
09:08:07  * scttnlsnquit (Ping timeout: 246 seconds)
09:11:26  * levelbotquit (Ping timeout: 240 seconds)
09:11:44  * levelbotjoined
09:13:17  * jcrugzzquit (Ping timeout: 240 seconds)
09:22:41  * kenan|afkchanged nick to kenansulayman
09:35:18  * funkytekquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
09:36:01  * funkytekjoined
09:39:11  * funkytekquit (Client Quit)
09:45:52  * ramitosjoined
09:47:14  * ramitosquit (Client Quit)
09:49:12  <levelbot>[npm] [email protected] <http://npm.im/blockchain-link>: Link - The Blockchain File Sharing Protocol (@tsavo)
09:49:53  * ramitosjoined
10:21:31  * tarrudajoined
10:24:26  * jcrugzzjoined
10:33:11  * frankblizzardjoined
10:35:43  <levelbot>[npm] [email protected] <http://npm.im/blockchain-link>: Link - The Blockchain File Sharing Protocol (@tsavo)
10:38:12  <levelbot>[npm] [email protected] <http://npm.im/blockchain-link>: Link - The Blockchain File Sharing Protocol (@tsavo)
10:38:13  * paulfryzeljoined
10:42:30  * paulfryzelquit (Ping timeout: 245 seconds)
10:50:36  * _Contrachanged nick to contraha_
10:50:40  * kenansulaymanchanged nick to kenan|afk
11:04:35  * scttnlsnjoined
11:08:59  * scttnlsnquit (Ping timeout: 250 seconds)
11:12:37  * stagasquit (Ping timeout: 248 seconds)
11:41:40  * contraha_changed nick to _Contra
11:46:26  * jcrugzzquit (Ping timeout: 264 seconds)
12:15:54  * frankblizzardpart
12:44:32  * jcrugzzjoined
13:05:32  * scttnlsnjoined
13:10:11  * scttnlsnquit (Ping timeout: 260 seconds)
13:32:25  * kenan|afkchanged nick to kenansulayman
13:38:06  * frankblizzardjoined
13:42:39  * frankblizzardquit (Ping timeout: 240 seconds)
13:54:10  <levelbot>[npm] [email protected] <http://npm.im/buffer-prefix-range>: Easily define lexicographical ranges of byte strings using a prefix. Can be used to define ranges for queries in leveldb or similar databases (@tarruda)
13:59:08  * dguttmanquit (Quit: dguttman)
14:05:04  * scttnlsnjoined
14:05:08  * scttnlsnquit (Remote host closed the connection)
14:19:27  * frankblizzardjoined
14:38:56  * daviddiasjoined
14:43:43  * scttnlsnjoined
15:00:08  * frankblizzardquit (Remote host closed the connection)
15:00:36  * frankblizzardjoined
15:04:47  * frankblizzardquit (Ping timeout: 240 seconds)
15:09:21  * jjmalinajoined
15:34:21  * thlorenzjoined
15:39:56  * tmcwjoined
15:41:01  * scttnlsnquit (Quit: Leaving...)
15:42:08  * paulfryzeljoined
15:43:21  * daviddia_joined
15:45:03  * daviddiasquit (Ping timeout: 240 seconds)
15:46:19  * paulfryzelquit (Ping timeout: 250 seconds)
15:46:29  * daviddia_quit (Read error: No route to host)
15:46:45  * daviddiasjoined
15:58:52  * dguttmanjoined
16:00:58  * dguttmanquit (Client Quit)
16:01:09  * jcrugzzquit (Ping timeout: 265 seconds)
16:06:33  * tmcwquit (Remote host closed the connection)
16:07:08  * tmcwjoined
16:09:01  * daviddiasquit (Remote host closed the connection)
16:09:32  * daviddiasjoined
16:09:46  * mikealquit (Quit: Leaving.)
16:11:37  * tmcwquit (Ping timeout: 252 seconds)
16:15:03  * tmcwjoined
16:18:55  * esundahljoined
16:23:51  * jcrugzzjoined
16:30:44  * daviddiasquit (Remote host closed the connection)
16:31:13  * daviddiasjoined
16:35:45  * daviddiasquit (Ping timeout: 245 seconds)
16:35:54  * timoxleyjoined
16:37:19  * jerrysvjoined
16:39:13  * mcavagejoined
16:42:46  * paulfryzeljoined
16:45:14  <levelbot>[npm] [email protected] <http://npm.im/mynpm>: your very own npm registry (@jmgunn87)
16:47:38  * paulfryzelquit (Ping timeout: 264 seconds)
16:52:02  * ramitosquit (*.net *.split)
17:01:27  * _Contrachanged nick to contraha_
17:09:40  * daviddiasjoined
17:11:24  * daviddia_joined
17:14:55  * daviddiasquit (Ping timeout: 245 seconds)
17:16:43  * daviddia_changed nick to daviddias
17:18:25  * jerrysv_joined
17:20:36  * mikealjoined
17:21:39  * jerrysvquit (Ping timeout: 250 seconds)
17:21:50  * contraha_changed nick to contrahax
17:22:00  * contrahaxchanged nick to contra
17:33:25  * mikealquit (Quit: Leaving.)
17:35:19  * mikealjoined
17:36:21  * timoxleyquit (Remote host closed the connection)
17:38:27  * contraquit
17:38:53  * timoxley_joined
17:38:57  * contrajoined
17:40:20  * tmcwquit (Remote host closed the connection)
17:40:56  * tmcwjoined
17:41:37  * contraquit (Client Quit)
17:41:57  * contrahaxjoined
17:43:19  * timoxley_quit (Ping timeout: 250 seconds)
17:45:50  * tmcwquit (Ping timeout: 264 seconds)
17:50:20  * daviddiasquit (Remote host closed the connection)
17:50:52  * daviddiasjoined
17:54:42  * daviddia_joined
17:54:55  * daviddiasquit (Ping timeout: 245 seconds)
17:58:59  * tmcwjoined
18:06:01  * mikealquit (Quit: Leaving.)
18:07:26  * tarrudaquit (Ping timeout: 264 seconds)
18:09:07  * daviddia_quit (Remote host closed the connection)
18:12:04  * jerrysv_changed nick to jerrysv
18:27:54  * briancjoined
18:28:04  <brianc>hi
18:30:28  * thlorenzquit (Remote host closed the connection)
18:31:47  <brianc>I'm having a strange issue with levelup on node
18:31:55  <brianc>I have a 1.5 million "row" database
18:32:03  <brianc>and i'm creating a read stream starting in the middle of it
18:32:16  <brianc>it seems to peg the CPU to 100% and stay there
18:32:29  <brianc>I'm using leveldown-hyper - not sure if that's the culprit
18:33:03  <brianc>some sort of tight loop or something because the node process basically becomes unresponsive
18:33:23  <brianc>it's spinning so hard on the CPU it takes like 60-70 seconds to make an http request
18:34:37  * daviddiasjoined
18:35:20  * daviddia_joined
18:35:45  * daviddiasquit (Read error: No buffer space available)
18:38:41  * daviddia_quit (Remote host closed the connection)
18:39:09  * thlorenzjoined
18:39:18  * mcavagequit (Remote host closed the connection)
18:39:20  * daviddiasjoined
18:39:48  * jerrysv_joined
18:39:55  * daviddiasquit (Read error: Connection reset by peer)
18:40:13  * daviddiasjoined
18:40:42  * mcavagejoined
18:40:57  <brycebaril>brianc: that's definitely strange, I've not seen that sort of thing with similarly sized dbs :(
18:41:06  <brianc>yah i'm still digging into it a bit
18:41:47  <brianc>if i read the stream slower (set a timeout between every call to read()) processor usage eventually drops back down
18:42:11  <brianc>it still pegs to 100% for the first 5-10 seconds...I guess while the reader buffers the first part into memory or something
18:42:25  * jerrysvquit (Ping timeout: 245 seconds)
18:43:05  <brianc>yah something is definitely screwed up here
18:43:05  <brycebaril>brianc: how big are your records on average?
18:43:24  <rescrv>brianc: post the "LOG" file from your leveldb directory
18:43:28  <brianc>a key with usually around a 1000 element array for the value
18:43:53  <brianc>and each array element is a json object of a few keys with guids for values
18:44:06  * daviddiasquit (Remote host closed the connection)
18:44:40  <brianc>there is a LOG and a LOG.old
18:45:00  <rescrv>cat LOG.old LOG > LOG.new # and post LOG.new
18:45:03  * daviddiasjoined
18:46:24  <brianc>https://gist.github.com/brianc/8044169
18:46:39  <brianc>not a lot of action going on there
18:51:26  * jxsonjoined
18:51:46  <brianc>strange...it slows down at a certain point in the key space
18:52:00  <brianc>and cpu remains pegged at 100%
18:52:04  <rescrv>are you reading and writing?
18:52:19  * tmcwquit
18:52:29  <brianc>i am reading the stream as fast as I can and just console.log the keys
18:52:37  <brianc>and its only able to read 2-3 keys a second
18:52:52  <brianc>in normal usage yes i'm reading and writing but at this point i'm just logging the keys
18:53:22  <brianc>memory is at like 3 gigabytes
18:53:46  <rescrv>how many ssts are there?
18:54:23  * tmcwjoined
18:54:25  <brianc>800
18:54:51  <brianc>it's strange it only happens in this section of the key space
18:55:01  <brianc>the values aren't much larger than in other areas of the database
18:55:16  <rescrv>is that part of the key space dense?
18:55:38  <brianc>the keys are 10 digit numbers
18:55:44  <brianc>but they're utf-8 strings
18:55:57  <brianc>and they're incrementing by 1-5
18:56:13  <brianc>so like....3073030038, 3073030039, 3073030041
18:57:53  <brianc>okay so another experiment...i set a timeout so I only read 1 row every 100ms. STILL pegs the CPU to 100% and uses 3 gigs of ram
18:58:36  <ggreer>I know very little about your problem, but if I were you, I'd probably pull out a profiler at this point. if you're on a mac, maybe instruments.app?
18:58:40  <ggreer>if linux, gprof
18:59:15  <rescrv>brianc: can you dump the db to show its structure?
18:59:27  <brianc>rescrv how do I do that?
19:01:00  <rescrv>can you build hyperleveldb outside of node? There's a tool called "leveldbutil". If you cannot do that, see if level exposes the GetProperty API
19:01:41  <brianc>do you think this is related more to hyperlevel than to leveldb?
19:01:59  <brianc>Also - they're binary compatible on disk, right? Is there a way I can convert this db to a non "hyper" version to see if the problem is there as well?
19:02:11  * ednapiranhajoined
19:02:48  <rescrv>at this point I have no idea if it's hyperleveldb-specific. The dump util or GetProperty calls will give me more information to make a call like that.
19:03:13  <rescrv>Alternatively, you can just switch from level-hyper to level and it should "just work". It'll only be work to go back
19:04:05  <rescrv>I suspect it's not a hyperleveldb-specific problem and just related to your data size.
19:04:11  <brianc>strange
19:04:16  <brianc>I'm building hyperleveldb now
19:04:47  <brianc>well having some problems with autoreconf -i
19:05:45  <rescrv>from the sound of it, each key is more than 128KB by itself (figure 32B per uuid, 4 uuids per element, 1000 elements per value)
19:06:12  <rescrv>I have no clue why it's using so much CPU, but big objects make iteration more costly
19:06:33  <brianc>they keys are small - just 10 bytes
19:06:39  <brianc>they values are larger
19:07:00  <rescrv>but it reads the key/value together and compares across them for iterators
19:07:23  <rescrv>so reading the next item from a stream could read many times more data from disk
19:07:37  <rescrv>and that's not specific to hyperleveldb
19:07:42  * ednapiranhaquit (Quit: Leaving...)
19:07:44  <brianc>dang
19:07:48  * timoxleyjoined
19:08:20  <brianc>so maybe leveldb not a good fit for this?
19:08:25  <rescrv>the good news is it's something that could be improved within LevelDB
19:09:07  <rescrv>if you're writing once, never overwriting, and never adding new keys, you could just use a flat file
19:09:20  <rescrv>if you can insert into leveldb in sorted order you'll see it work better too
19:09:38  <brianc>I've overwriting a lot
19:09:50  <brianc>i start w/ keys and blank values
19:09:57  <brianc>then itterate over the keyspace and update the values
19:09:58  <brianc>a few times
19:10:04  * daviddiasquit (Remote host closed the connection)
19:10:08  <brianc>incrementally building on the data gathered from the last iteration
19:10:24  <brianc>i never insert after the first time, only updates
19:10:28  * mcavagequit (Remote host closed the connection)
19:10:44  * daviddiasjoined
19:11:03  <brianc>okay i got hyperleveldb built, how do I dump this for you?
19:11:25  * mikealjoined
19:12:14  * timoxleyquit (Ping timeout: 240 seconds)
19:12:51  <rescrv>brianc: when you say values, you mean the key/uuid pairs, elements in the array, or the entire array?
19:13:26  <rescrv>"./leveldbutil dump /path/to/db-dir/MANIFEST-*"
19:13:46  <rescrv>it'll post parts of your values, so you may want to post it somewhere private
19:13:51  <brianc>when I say key I mean the 10 byte numeric key I am storing, and when i say values, I mean everything "not the key" which would be a JSON.stringify([{id: <guid>, score: int, }]) array of about 1000 items
19:14:07  <rescrv>and how much of that value changes on overwrite?
19:14:53  <brianc>at this point the value changes from having an empty value or a value of just "{}" to the whole thing
19:15:11  * daviddiasquit (Ping timeout: 250 seconds)
19:16:15  <rescrv>ok. so at the moment you have at most two versions of each key?
19:16:38  * daviddiasjoined
19:17:11  * daviddiasquit (Remote host closed the connection)
19:17:33  <brianc>yah
19:17:41  * daviddiasjoined
19:19:54  * ednapiranhajoined
19:29:31  * mcavagejoined
19:29:50  * mcavagequit (Read error: Connection reset by peer)
19:30:10  * mcavagejoined
19:38:58  * tarrudajoined
20:08:48  * timoxleyjoined
20:13:25  * timoxleyquit (Ping timeout: 248 seconds)
20:13:30  * jxsonquit (Remote host closed the connection)
20:29:02  * ramitosjoined
20:37:54  * chandru_injoined
20:38:24  <chandru_in>If I insert large values with random keys, will leveldb
20:38:39  <chandru_in>constantly move subsequent data blocks in the file?
20:45:56  * ednapiranhaquit (Quit: Leaving...)
20:48:40  <levelbot>[npm] [email protected] <http://npm.im/riffle>: Forward and reverse iteration of a Strata b-tree. (@bigeasy)
20:49:13  * jmartinsjoined
20:53:35  <kenansulayman>chandru_in append-only
20:53:38  * funkytekjoined
20:54:19  <kenansulayman>depends on the layer the file is in tho
20:55:16  <rescrv>chandru_in: it'll rewrite each block to move it up the levels
20:55:23  <chandru_in>kenansulayman, This may be naive but table_format says blocks are stored in sorted order of keys. If I insert a key that lies in between existing blocks, shouldn't they move
20:55:43  <kenansulayman>rescrv ^
20:55:43  <rescrv>what will happen is you'll do a merge sort on a set of tables to create a new set of tables
20:55:46  <kenansulayman>hah
20:56:01  <rescrv>so you rewrite, but once a table is created it will never move
20:56:32  <chandru_in>How does a key look up work?
20:56:34  <rescrv>that being said, large values with random keys usually can be written in a better way
20:57:09  <rescrv>in the most general of terms, you look for the key in each level until you find a level that has it
20:57:33  <rescrv>the search within a level consists of finding the SST that could hold it, and then using the index at the end of the table to know the offset to look for the key
20:59:55  <chandru_in>If the same key is updated, can the updates end up in multiple young level files?
21:00:49  <rescrv>yes
21:01:21  <chandru_in>Thanks a lot, rescrv
21:01:45  <chandru_in>So if I have a large number of levels, won't reads be slowed proportionally?
21:05:57  * esundahlquit (Remote host closed the connection)
21:06:26  * esundahljoined
21:06:52  <rescrv>chandru_in: at some point, yes. bloom filters and cache will take the majority of the hit though.
21:07:16  <rescrv>if you have 10x the data as memory, you'll only really read from disk for things in the last level (in theory)
21:08:39  <chandru_in>rescrv, thanks for all the help. Is there a better place to understand leveldb's implementation other than https://leveldb.googlecode.com/git/doc/impl.html
21:08:53  <chandru_in>short of reading the code
21:09:05  <rescrv>you pre-emptively shot down my response
21:09:31  <rescrv>best bet is to ask questions of the list. The BigTable paper gives a good overview of an alternative design that inspired leveldb
21:09:34  * timoxleyjoined
21:10:36  <chandru_in>rescrv, thanks a lot.
21:11:01  * esundahlquit (Ping timeout: 248 seconds)
21:14:07  * timoxleyquit (Ping timeout: 252 seconds)
21:14:15  * jerrysv_changed nick to jerrysv
21:24:51  * chandru_inpart ("Ex-Chat")
21:38:31  * abstractjquit (Ping timeout: 246 seconds)
21:38:56  * abstractjjoined
21:38:59  * abstractjquit (Changing host)
21:38:59  * abstractjjoined
21:40:38  * Industrialjoined
21:41:12  <Industrial>Hi. Say I save a posts:1:title foo, how would I get a list of all posts? Trying to get a REST index route implemented..
21:41:52  * funkytekquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
21:53:11  * funkytekjoined
21:54:01  <chilts>createReadStream using 'posts:' and 'posts:~' as the start/end should get you all posts
21:54:15  * funkytekquit (Client Quit)
21:54:27  <chilts>also, perhaps read up and consider using \xFF as your separator, rather than ':'
21:54:30  <chilts>Industrial: ^^
21:54:46  <chilts>of course, that also depends on how much control you have as to what goes into the keys
22:10:09  * timoxleyjoined
22:14:15  * timoxleyquit (Ping timeout: 240 seconds)
22:19:19  * ralphtheninjajoined
22:19:24  * funkytekjoined
22:21:24  * mcavagequit (Ping timeout: 246 seconds)
22:21:58  * mcavagejoined
22:22:00  * daviddiasquit (Ping timeout: 245 seconds)
22:23:13  * funkytekquit (Client Quit)
22:35:38  * tarrudaquit (Ping timeout: 264 seconds)
22:47:31  * paulfryzeljoined
22:51:51  * paulfryzelquit (Ping timeout: 250 seconds)
22:53:55  * timoxleyjoined
22:57:52  <jiffe99>can multiple threads write to the same db simultaneously or do I need to implement locking around that?
23:04:45  * jxsonjoined
23:07:59  * funkytekjoined
23:08:06  * funkytekquit (Client Quit)
23:17:25  * funkytekjoined
23:24:14  <kenansulayman>jiffe99 it wouldn't even work w/ locking w/o closing the db
23:24:52  <kenansulayman>you might consider level-lmdb / uberlevel for multithread environments
23:25:14  <rescrv>kenansulayman: false! leveldb is threadsafe out of the box, just as lmdb is.
23:25:23  <kenansulayman>rescrv sorry
23:25:40  <kenansulayman>I never managed to get it running though?
23:25:46  <kenansulayman>At least the node bindings
23:26:15  <kenansulayman>It crashes when trying to open a db from another cluster instance
23:31:21  * thlorenzquit (Remote host closed the connection)
23:38:46  <rescrv>kenansulayman: I think we just implemented the question differently. He asked about threads writing to the same db. I interpreted it as the "instance" returned by open. You interpreted it as multiple "opens". jiffe99: yes and no
23:41:43  <jiffe99>no I am using a single opened instance
23:42:01  <kenansulayman>rescrv yes, indeed. I am talking about opening a database again — not reusing the instance
23:42:48  <rescrv>but jiffe99 needs a single instance
23:43:12  <kenansulayman>jiffe99 are you in node.js-land or c++?
23:43:27  <jiffe99>kenansulayman: c++
23:43:38  <kenansulayman>that's rescrv's department then =)
23:49:12  * ramitosquit (Quit: Textual IRC Client: www.textualapp.com)