00:01:26  * tmcwquit (Ping timeout: 240 seconds)
00:04:14  <Raynos>mikeal: correction, learnboost is not to be trusted
00:04:23  <Raynos>this is why you rewrite something like engine.io
00:09:32  * ednapiranhaquit (Remote host closed the connection)
00:18:27  * thlorenzjoined
00:26:45  * thlorenzquit (Ping timeout: 248 seconds)
00:46:11  <dscape>guys
00:46:13  <dscape>newbie halp
00:46:24  <dscape>OpenError: IO error: lock ./leveldb/LOCK
00:46:34  <dscape>just one process running
00:46:39  <dscape>no operations at the same time
00:46:58  <rvagg>are you *sure* there's only one process? and is this on a plain filesystem or something magical like vagrant?
00:47:26  <dscape>rvagg: yeah only one process
00:47:33  * thlorenzjoined
00:47:39  <dscape>everything was working until i used code by mikeal
00:47:42  <dscape>basically :P
00:47:51  <dscape>im going to print that in a tshirt
00:47:52  <rvagg>ok, so check that the db isn't being opened twice by the same process
00:48:15  <rvagg>if it is, is it being closed first before reopened?
00:48:22  <dscape>got it
00:48:32  <rvagg>can't do level('/path/') more than once without a .close() in between
00:48:33  <dscape>i think that's the case
00:48:44  <dscape>is it good practice to use a cazilions databases?
00:48:45  <dscape>:P
00:48:46  <rvagg>.. but you shouldn't have to do that anyway, just one open per process should be fine
00:49:00  <dscape>or is that a bad idea too
00:49:01  <dscape>?
00:49:07  <mikeal>haha
00:49:20  <rvagg>depends on the reason, you can but you probably should try and find a way to use a single db
00:49:23  <rvagg>(like using sublevel)
00:49:27  <mikeal>dscape: put the core you're running somewhere
00:49:30  <rvagg>that way they can share cache
00:50:27  <dscape>https://github.com/dscape/magic-pandas-and-unicorns
00:50:44  <dscape>mikeal: ^
00:50:46  <dscape>codez
00:51:18  <dscape>you making a mess
00:51:18  <dscape>https://github.com/dscape/magic-pandas-and-unicorns/blob/master/lib/scraper.js#L4
00:53:15  * thlorenz_joined
00:53:20  <mikeal>i wouldn't do that
00:53:27  <mikeal>instantiate a database in the module scope
00:53:33  <mikeal>it could potentially get called twice
00:54:02  <mikeal>you've got globals all over this like you're writing browser code :)
00:57:20  <dscape>mikeal: i could easily say that scraping a shitty website no one cares about
00:57:25  <dscape>quality is not very important :P
00:57:27  <dscape>eheh
00:57:31  * tmcwjoined
00:59:02  <mikeal>it's not just quality, it's increasing your ability to make silly mistakes
00:59:09  <mikeal>i've written code like this before thinking the same thing
00:59:13  <mikeal>and always regret it
01:01:25  * thlorenz_quit (Ping timeout: 248 seconds)
01:01:50  * tmcwquit (Ping timeout: 240 seconds)
01:04:25  * dguttmanquit (Quit: dguttman)
01:04:37  * mikealquit (Quit: Leaving.)
01:04:44  * timoxleyquit (Remote host closed the connection)
01:04:49  <dscape>mikeal true, especially cause it just happened
01:04:49  <dscape>rofl
01:19:14  * mikealjoined
01:27:10  * jxsonquit (Remote host closed the connection)
01:28:18  * thlorenz_joined
01:32:37  * thlorenz_quit (Ping timeout: 246 seconds)
01:46:26  * mikealquit (Quit: Leaving.)
01:46:51  * levelbotquit (Remote host closed the connection)
01:47:06  * levelbotjoined
02:02:02  * jcrugzzquit (Read error: Connection reset by peer)
02:02:28  * jcrugzzjoined
02:03:59  * soldairquit (Ping timeout: 250 seconds)
02:18:19  * gwenbelljoined
02:28:06  * tmcwjoined
02:28:42  * thlorenz_joined
02:32:40  * tmcwquit (Ping timeout: 256 seconds)
02:32:53  * thlorenz_quit (Ping timeout: 240 seconds)
02:42:48  * gwenbellquit (Quit: Lost terminal)
03:14:13  * thlorenzquit (Read error: Connection reset by peer)
03:14:30  * thlorenzjoined
03:29:06  * thlorenz_joined
03:29:56  * soldairjoined
03:33:25  * thlorenz_quit (Read error: Operation timed out)
03:36:15  * thlorenzquit (Remote host closed the connection)
03:49:45  * dominictarrjoined
03:59:30  * thlorenzjoined
04:03:51  * thlorenzquit (Ping timeout: 260 seconds)
04:17:08  * julianduquequit (Quit: leaving)
04:33:42  * dominictarrquit (Quit: dominictarr)
04:59:54  * thlorenzjoined
05:03:34  * jxsonjoined
05:04:32  * thlorenzquit (Ping timeout: 256 seconds)
05:45:27  * st_lukejoined
05:58:59  * mikealjoined
05:59:18  * mikealquit (Client Quit)
06:00:22  * thlorenzjoined
06:01:37  * timoxleyjoined
06:04:38  * thlorenzquit (Ping timeout: 240 seconds)
06:07:01  * st_lukequit (Remote host closed the connection)
06:27:37  * jxsonquit (Remote host closed the connection)
06:28:00  * mikealjoined
06:33:02  * keslajoined
07:00:48  * thlorenzjoined
07:05:00  * thlorenzquit (Ping timeout: 245 seconds)
07:09:03  * wolfeidauquit (Remote host closed the connection)
07:19:52  * mikealquit (Quit: Leaving.)
07:21:47  * timoxleyquit (Remote host closed the connection)
07:41:41  * Acconutjoined
07:41:53  * Acconutquit (Client Quit)
07:59:05  * mikealjoined
08:01:09  * thlorenzjoined
08:03:05  * wolfeidaujoined
08:05:26  * thlorenzquit (Ping timeout: 240 seconds)
08:51:32  * dominictarrjoined
08:52:57  * dominictarrquit (Client Quit)
08:59:01  <levelbot>[npm] [email protected] <http://npm.im/levelup-cursor>: levelup stream cursor (@ramitos)
09:01:38  * thlorenzjoined
09:05:40  * thlorenzquit (Ping timeout: 240 seconds)
09:10:01  <levelbot>[npm] [email protected] <http://npm.im/range-index>: levelup based range index (@ramitos)
09:13:07  * dubbanjoined
09:14:23  * dominictarrjoined
09:18:33  <levelbot>[npm] [email protected] <http://npm.im/range-index>: levelup based range index (@ramitos)
09:57:55  * jcrugzzquit (Ping timeout: 256 seconds)
10:02:02  * thlorenzjoined
10:06:34  * thlorenzquit (Ping timeout: 256 seconds)
10:14:10  * keslaquit (Ping timeout: 245 seconds)
10:18:26  * wolfeidauquit (Remote host closed the connection)
10:26:03  * wolfeidaujoined
10:26:49  * jcrugzzjoined
10:27:43  * Acconutjoined
10:27:46  * Acconutquit (Client Quit)
10:33:01  * jcrugzzquit (Ping timeout: 246 seconds)
10:39:03  * wolfeidauquit (Remote host closed the connection)
10:59:53  * dubbanquit (Remote host closed the connection)
11:02:27  * thlorenzjoined
11:06:45  * thlorenzquit (Ping timeout: 248 seconds)
11:36:38  * timoxleyjoined
11:54:43  * thlorenzjoined
12:02:51  * thlorenz_joined
12:07:27  * thlorenzquit (Remote host closed the connection)
12:07:41  * thlorenz_quit (Ping timeout: 256 seconds)
12:12:30  * thlorenzjoined
12:41:31  <levelbot>[npm] [email protected] <http://npm.im/dscape-mind-explosion-database>: DATABASE FOR DSCAPE TO TEACH CHILDREN! (@mikeal, @dscape)
12:46:03  * thlorenzquit (Ping timeout: 260 seconds)
12:47:49  * thlorenzjoined
12:48:50  * nialloZquit (Ping timeout: 264 seconds)
12:49:04  * nialloZjoined
12:59:36  * thlorenzquit (Remote host closed the connection)
13:03:18  * thlorenzjoined
13:07:59  * thlorenzquit (Ping timeout: 260 seconds)
13:08:54  <rvagg>really? that's continuing development?
13:09:10  * jondelamottequit (Ping timeout: 245 seconds)
13:23:18  * ednapiranhajoined
13:38:37  * soldairquit (Ping timeout: 250 seconds)
13:39:51  * thlorenzjoined
13:41:53  * ednapiranhaquit (Remote host closed the connection)
13:50:47  * tmcwjoined
14:36:59  <dscape>rvagg: of course
14:46:17  * mikealquit (Read error: Connection reset by peer)
14:59:08  * dguttmanjoined
15:03:10  * soldairjoined
15:07:52  * jondelamottejoined
15:17:33  <dguttman>anyone know of a module like level-live-stream that gives updates with the old/previous value?
15:21:29  * fallsemojoined
15:25:54  <soldair>dguttman: there are modules for immutable/versioned keys but i don't know of any that simply do a get before a set like that
15:27:13  * mikealjoined
15:27:32  <mikeal>haha
15:31:31  <dguttman>soldair: thanks, yea, figured it would require monkeying with "put" or something like that
15:32:13  <brycebaril>that might be a good plugin for level-version -- that way the old value lookup could be done after the new value is put
15:32:41  <brycebaril>though there's definitely an overhead for level-version, so I don't know what the net gain would be
15:32:49  * mikealquit (Quit: Leaving.)
15:35:56  <soldair>im so happy right now im getting close to a viable js only leveldown. ill call for code review/slamming soon
15:51:11  * jerrysvjoined
16:12:57  * mikealjoined
16:21:49  * paulfryzeljoined
16:42:48  * thlorenzquit (Remote host closed the connection)
16:45:29  * jxsonjoined
17:12:16  * thlorenzjoined
17:27:08  * paulfryzelquit (Remote host closed the connection)
17:27:48  * paulfryzeljoined
17:30:05  * Acconutjoined
17:35:19  * Acconutquit (Quit: Acconut)
17:40:47  * julianduquejoined
17:43:00  <levelbot>[npm] [email protected] <http://npm.im/abstract-store>: a module to abstract key/value storage away from the underlying technology (@phidelta)
17:44:30  <levelbot>[npm] [email protected] <http://npm.im/abstract-store>: a module to abstract key/value storage away from the underlying technology (@phidelta)
17:47:46  * dominictarrquit (Quit: dominictarr)
17:51:11  * jcrugzzjoined
18:01:39  * soldairquit (Ping timeout: 250 seconds)
18:14:06  * Acconutjoined
18:27:51  * julianduquequit (Ping timeout: 276 seconds)
18:33:30  <levelbot>[npm] [email protected] <http://npm.im/tik>: command line key/value store (@jarofghosts)
18:54:40  * soldairjoined
19:07:30  <levelbot>[npm] [email protected] <http://npm.im/readme-getter>: A module for fetching README html from npm modules. (@danfinlay)
19:16:36  * jmartinsjoined
19:26:00  <levelbot>[npm] [email protected] <http://npm.im/readme-getter>: A module for fetching README html from npm modules. (@danfinlay)
19:26:08  * paulfryz_joined
19:26:55  * paulfry__joined
19:29:36  * paulfryzelquit (Ping timeout: 276 seconds)
19:30:45  * paulfryz_quit (Ping timeout: 264 seconds)
19:31:15  * paulfry__quit (Ping timeout: 245 seconds)
19:38:16  * brianloveswordsquit (Excess Flood)
19:39:07  * brianloveswordsjoined
19:39:19  * thlorenztopic: Streichhölzer
19:46:11  * Acconutquit (Quit: Acconut)
19:56:07  * jxsonquit (Remote host closed the connection)
20:08:36  * ednapiranhajoined
20:12:14  * tmcwquit (Remote host closed the connection)
20:13:40  * mikealquit (Quit: Leaving.)
20:14:12  * keslajoined
20:21:14  * tmcwjoined
20:22:48  * tmcwquit (Remote host closed the connection)
20:27:53  * tmcwjoined
20:43:53  * mikealjoined
20:48:28  * mikealquit (Ping timeout: 264 seconds)
20:58:32  * jmartinsquit (Quit: Leaving)
21:01:27  * keslaquit (Remote host closed the connection)
21:01:45  * wolfeidaujoined
21:03:17  * mikealjoined
21:04:27  * paulfryzeljoined
21:27:29  * jxsonjoined
21:28:50  * jxsonquit (Remote host closed the connection)
21:30:25  * jxsonjoined
21:39:39  * brianloveswordsquit (Excess Flood)
21:41:07  * brianloveswordsjoined
21:51:31  * brianc_joined
21:51:36  <brianc_>hey
21:51:39  <brianc_>i have a questiona
21:51:44  <brianc_>about node + leveldb
21:52:24  <brianc_>how does it make sense to use leveldb as a database if you can only access it from a single process at a time? Seems like a pretty big bottleneck
21:54:04  <mbalho>node is a single process too (single threaded)
21:54:11  <brianc_>right
21:54:21  <brianc_>so to take advantage of multiple cores you run node w/ the cluster module
21:54:28  <brianc_>but you cannot access leveldb in that situation
21:54:32  <brianc_>unless each process has its own copy
21:54:35  <brianc_>which is slightly ridiculous
21:54:38  <soldair>mysql is a single process most of the time as well
21:55:29  <mbalho>brianc_: do you have a specific use case in mind?
21:55:55  <brianc_>an append & read only database of events
21:55:59  <brianc_>kinda like datomic
21:56:24  <mikeal>brianc_: you would use leveldb to write a database that people access over the network
21:56:30  <mbalho>brianc_: cool, you do you imagine 100% cpu saturation across all cpus? and do you need each process to be able to write to the db?
21:56:55  <brianc_>I don't imagine 100% cpu saturation across all cpus
21:57:11  <brianc_>but in a 16 core box 15 are gonna sit idle with leveldb
21:57:14  <mikeal>databases don't really saturate CPU, in my experience :)
21:57:25  <mikeal>databases tend to be disc bound
21:57:26  <soldair>in that case you should either partition our keyspace across a child process or use replication
21:57:37  <soldair>but yeah what he said =)
21:57:41  <jerrysv>mikeal: you and i live in very different worlds :)
21:59:06  * thlorenzquit (Remote host closed the connection)
22:00:25  <brianc_>is there a way to do a "put if not exists" in leveldb?
22:00:30  <levelbot>[npm] [email protected] <http://npm.im/level-user>: client side library for moving data over a level-socket (@maxogden)
22:01:28  <soldair>brianc_: you would have to get first or always insert a new row with a unique key suffix
22:01:59  <brianc_>soldair: so what happens if you do a get and then someone puts after your get but before your put?
22:02:20  <mbalho>you can do a batch (transaction)
22:02:20  <brianc_>aka how do you "lock" something with leveldb?
22:02:21  * jcrugzzquit (Ping timeout: 276 seconds)
22:02:35  <soldair>brianc_: its easy to stop concurrent writes to the same key because js is not threaded
22:03:06  <rvagg>mikeal has level-mutex now doesn't he?
22:03:24  <soldair>brianc_: just hold the keys being put in a js object
22:04:12  <soldair>rvagg: yeah that's a bit classier
22:04:41  <mikeal>yes i do :)
22:06:23  <soldair>rvagg: know the best place to read up on levels serialization format for keys/values in the ssts? I'm headed to the source but just looking for shortcuts.
22:06:43  <brianc_>I'm digging level-mutex
22:09:23  * no9quit (Read error: Operation timed out)
22:09:30  * mikealquit (Quit: Leaving.)
22:09:36  <rvagg>soldair: you could have a poke around in leveljs
22:09:39  <brianc_>and so to "query" leveldb the prescribed method is to iterate over a set of keys doing a map-reduce kinda thing?
22:09:41  <rvagg>soldair: it's partially implemented
22:10:03  <brianc_>and level-sublevel basically just appends a key to the front of your key so it can segment things into different "collections"?
22:10:05  <rvagg>brianc_: "query" involves clever organising your keys to start with
22:10:11  * mikealjoined
22:10:37  <rvagg>and yes, re sublevel
22:11:15  <brianc_>yeah I was thinking of doing something like <type>-<actual-key>-<datestamp> for an append only thing
22:11:42  <brianc_>so if you do a "put" with a key I append a date to the end so you never replace data, just append a new value
22:13:23  * no9joined
22:13:35  <brianc_>one last question: anyone experience corruption on their leveldb data? Any easy things to NOT do to avoid corruption?
22:15:18  * mcollinajoined
22:15:54  * keslajoined
22:17:52  <brianc_>I realize I phrased my question incorrectly
22:18:08  <brianc_>a shorter question: is there an FAQ about leveldb or a nice getting started guide anywhere?
22:20:49  * tmcwquit (Remote host closed the connection)
22:20:50  * keslaquit (Ping timeout: 245 seconds)
22:20:50  * julianduquejoined
22:24:36  <mikeal>brianc_: do you want a guide to the level stuff in node or leveldb in general?
22:24:36  <mbalho>brianc_: http://r.va.gg/presentations/sf.nodebase.meetup/#/ is one resource
22:24:57  <mikeal>yeah, that would be my suggestion as well
22:27:22  * mikealquit (Quit: Leaving.)
22:28:32  <brianc_>mostly level stuff
22:28:36  <brianc_>the node stuff I can read no problemo
22:28:54  <rvagg>http://dailyjs.com/2013/04/19/leveldb-and-node-1/ http://dailyjs.com/2013/05/03/leveldb-and-node-2
22:29:12  <brianc_>thanks y'all
22:29:36  <rvagg>covers keys a bit at the bottom of that second article
22:29:37  * thlorenzjoined
22:35:01  <brianc_><3
22:35:04  <brianc_>perfectamundo
22:39:04  * mikealjoined
22:39:28  * thlorenzquit (Ping timeout: 264 seconds)
22:39:37  * fallsemoquit (Quit: Leaving.)
22:40:38  * brianc_quit (Remote host closed the connection)
22:45:48  * mikealquit (Quit: Leaving.)
22:49:52  * mikealjoined
22:54:49  * fallsemojoined
23:03:34  * mcollinaquit (Remote host closed the connection)
23:04:11  * mcollinajoined
23:04:18  * mcollinaquit (Remote host closed the connection)
23:04:51  * mcollinajoined
23:06:18  * mcollinaquit (Remote host closed the connection)
23:16:49  * fallsemoquit (Quit: Leaving.)
23:19:23  * paulfryzelquit (Remote host closed the connection)
23:22:52  * wolfeidauquit (Remote host closed the connection)
23:23:20  * wolfeidaujoined
23:27:55  * wolfeidauquit (Ping timeout: 245 seconds)
23:31:20  * tmcwjoined
23:34:26  * mikealquit (Quit: Leaving.)
23:34:51  * thlorenzjoined
23:35:52  * thlorenz_joined
23:35:52  * thlorenzquit (Read error: Connection reset by peer)
23:35:58  * tmcwquit (Ping timeout: 246 seconds)
23:37:02  * mikealjoined
23:37:23  * jerrysvquit (Remote host closed the connection)