17:38:20  * kenansulaymantopic: http://logs.nodejs.org/ — http://r.va.gg/2013/11/leveldown-v0.10-managing-gc-in-native-v8-programming.html
17:42:22  * brycebariljoined
17:46:03  <levelbot>[npm] [email protected] <http://npm.im/npmd>: distributed npm client (@dominictarr)
17:50:51  * jcrugzzquit (Ping timeout: 252 seconds)
17:59:52  * tmcwquit (Remote host closed the connection)
18:05:16  * kevinswiberjoined
18:21:07  * groglogicquit (Ping timeout: 264 seconds)
18:22:27  * thlorenzjoined
18:22:53  * groglogicjoined
18:27:20  * leveldb-noobiejoined
18:28:00  <leveldb-noobie>Hi Folks, I have a question regarding leveldb. Does it have MVCC?
18:29:20  <rescrv>leveldb-noobie: It's a key-value store. It's certainly possible to implement MVCC on top, but it's not included with it.
18:29:32  <leveldb-noobie>ok
18:30:58  * tmcwjoined
18:31:01  <leveldb-noobie>Also, I have a requirement where I need to bulk load the data into leveldb to generate uniqueID's. So, if the key already exists, I need to ignore it; otherwise, insert that key and generate a unique ID.
18:31:49  <leveldb-noobie>Bulk data will come in chunks of 50-70 million records; and the key-value store should scale upto a billion values.
18:31:59  <leveldb-noobie>I am wondering how level is going to perform.
18:32:03  <leveldb-noobie>leveldb*
18:32:20  * tmcwquit (Read error: Connection reset by peer)
18:32:34  * tmcwjoined
18:32:58  <ryan_ramage>leveldb-noobie: I found this informative: https://gist.github.com/maxogden/6551333
18:35:29  * Punnajoined
18:35:34  * thlorenzquit (Ping timeout: 246 seconds)
18:36:15  * Punnaquit (Client Quit)
18:39:01  <leveldb-noobie>Thanks, ryan_ramage! My requirement is slightly different -- I need to look up the data before insert it. It is basically a "insert if not exists condition'
18:39:33  * insertcoffeequit (Read error: Operation timed out)
18:40:20  * jcrugzzjoined
18:46:42  <ogd>leveldb-noobie: see https://github.com/maxogden/dat/blob/master/lib/commands.js#L468
18:46:57  <ogd>leveldb-noobie: warning: that code is kind of complex, needs refactoring
18:47:30  <ogd>leveldb-noobie: generally speaking though i think dat has a lot of the same feature you have mentioned
18:49:20  <ogd>leveldb-noobie: but the gist linked earlier has the basis for what you need, adding the insert if not exists logic is just another preprocessing step
18:50:20  * jxsonjoined
18:50:31  * jcrugzzquit (Ping timeout: 272 seconds)
18:52:06  * tarrudaquit (Remote host closed the connection)
18:55:28  * jcrugzzjoined
19:13:04  * wolfeidaujoined
19:32:23  * thlorenzjoined
19:33:38  * wolfeidauquit (Read error: Connection reset by peer)
19:36:23  * esundahlquit (Remote host closed the connection)
19:36:50  * esundahljoined
19:38:37  * tarrudajoined
19:41:49  * esundahlquit (Ping timeout: 272 seconds)
19:42:07  * leveldb-noobiequit (Ping timeout: 250 seconds)
19:45:41  * thlorenzquit (Ping timeout: 248 seconds)
19:51:55  * esundahljoined
19:59:12  * soldairjoined
20:27:13  * groglogicquit (Ping timeout: 245 seconds)
20:40:05  * jcrugzzquit (Ping timeout: 272 seconds)
20:42:18  * thlorenzjoined
20:44:22  * thlorenz_quit (Remote host closed the connection)
20:48:45  * ramitosjoined
20:55:34  * thlorenzquit (Ping timeout: 246 seconds)
20:59:27  * enosjoined
21:10:57  * jxsonquit (Remote host closed the connection)
21:14:28  <ogd>someone should write http://docs.aws.amazon.com/redshift/latest/dg/c_Window_functions.html for leveldb :D
21:15:18  * thlorenzjoined
21:15:35  * kevinswiberquit (Remote host closed the connection)
21:19:23  <brycebaril>ogd: timestream has a lot of that stuff for ordered series, it's in small enough pieces you might be able to just apply some of those parts to generic collections
21:21:24  <brycebaril>ogd: if not it can always be split into smaller pieces :)
21:23:49  * thlorenzquit (Ping timeout: 248 seconds)
21:24:28  <ogd>brycebaril: nice
21:24:32  <ogd>brycebaril: check out this query https://gist.github.com/fredbenenson/7566727
21:24:47  <ogd>brycebaril: i THINk what this means: SUM(1) OVER(PARTITION BY month ORDER BY pledge_count DESC ROWS UNBOUNDED PRECEDING)
21:24:58  <ogd>is split data by month
21:25:04  <ogd>then sort it by pledge count
21:25:23  <ogd>then for each item in each month get the year to date sum
21:25:30  <brycebaril>ogd: that is **exactly** the type of stuff timestreamdb is for :)
21:26:05  <brycebaril>it has built-in time interval aggregation
21:26:46  * tarrudaquit (Quit: WeeChat 0.4.2)
21:26:47  <ogd>brycebaril: can you explain that a little more?
21:27:10  * tarrudajoined
21:28:04  <brycebaril>a stream of data by day inputstream.sum('month') >> a stream of that data summed by month
21:28:26  <ogd>oh ok
21:28:56  <ogd>brycebaril: have you looked into parallelization ever?
21:29:04  <brycebaril>it uses leveldb with timestamped data (using level-version) and then puts statistical decorators on readstream
21:29:20  <ogd>dude why have i never read through all of your npm packages
21:29:57  <brycebaril>it's definitely experimental but I plan on spending lots of time on making it all kinds of awesome
21:30:09  <ogd>well im doing very similar stuff with dat
21:30:58  <ogd>like... i forgot about level-version
21:31:06  <brycebaril>yeah I've definitely written some stuff you could probably use, like array-pivot + stats-lite
21:31:51  <ogd>so as my vision of dat core emerges (haha) I am realizing the goal needs to be an awesome api for versioning + real time sync
21:32:06  <ogd>originally i was mimicking git in that you would make discrete dat commands from the command line
21:32:23  <ogd>but now im realizing that dat needs to be more of a daemon that you interact with over a procotol like couchdb
21:32:51  <ogd>since if you are doing a write in dat from the command line you cant open it from another process
21:33:41  <brycebaril>interesting
21:33:48  * matehatquit (Quit: Leaving...)
21:34:17  <ogd>and i cant get around certain brute force things like if you have a giant CSV and you import it, then change one line and import it again -- you have to process the whole damn thing again
21:34:37  <ogd>so i need to have nice apis for streaming progress
21:34:47  <brycebaril>yeah
21:36:24  <brycebaril>are you going to be at nodesummit?
21:36:30  <ogd>nope
21:36:39  <ogd>not a fan of that event
21:37:01  <ogd>i might be on the east coast then too
21:37:20  <brycebaril>I wasn't going to (also not really a fan of it) but the company that acquihired me through buying ravenwall is sending me
21:37:30  <brycebaril>sounds like we won't be able to chat dat in person then :/
21:38:01  <ogd>brycebaril: whats that company called btw?
21:38:56  <brycebaril>Concurix
21:39:12  <brycebaril>It's still pretty small, 8 of us
21:39:52  <ogd>ah interesting, who are the main clients (if its public info)?
21:41:27  <brycebaril>still trying to get a foot-hold, I don't think our current client list is public.
21:52:09  * thlorenzjoined
21:53:32  <levelbot>[npm] [email protected] <http://npm.im/level-relation>: relations and joins between sublevels (@stagas)
21:54:53  <ogd>brycebaril: question about https://github.com/brycebaril/level-bufferstreams, do you think it would be even faster to just do this in c++?
22:04:05  * jxsonjoined
22:05:21  * thlorenzquit (Ping timeout: 240 seconds)
22:06:08  * stagasjoined
22:19:04  * mcavagejoined
22:33:51  * tarrudaquit (Read error: Connection reset by peer)
22:40:48  <brycebaril>ogd: probably
22:42:35  <ogd>brycebaril: kinda reminds me of multiget
22:42:51  <ogd>brycebaril: also https://github.com/facebook/rocksdb/wiki/Rocksdb-Architecture-Guide#gets-iterators-and-snapshots
22:42:57  <ogd>brycebaril: apparently they implemented multiget
22:47:01  <brycebaril>ogd: I've been so slammed I haven't even had a chance to look at rocksdb yet
22:50:48  * stagasquit (Ping timeout: 265 seconds)
22:54:03  <ogd>brycebaril: no worries its nothing revolutonary, just more competition which is always good
22:57:04  * paulfryzelquit (Read error: Connection reset by peer)
22:57:42  * paulfryzeljoined
23:02:05  * thlorenzjoined
23:02:11  * thlorenz_joined
23:02:59  * thlorenz_quit (Remote host closed the connection)
23:15:21  * thlorenzquit (Ping timeout: 246 seconds)
23:16:45  * kenansulaymanjoined
23:26:13  * tmcwquit (Remote host closed the connection)
23:27:31  <levelbot>[npm] [email protected] <http://npm.im/meatspace-jamon>: Threaded chat with LevelDB (@ednapiranha)
23:43:07  <rescrv>ogd: is their multi-get anything smarter than just iterated get on the same version?
23:45:18  * jcrugzzjoined
23:50:29  * thlorenzjoined
23:51:44  * thlorenz_joined
23:51:44  * thlorenzquit (Read error: Connection reset by peer)
23:53:01  * jxsonquit (Remote host closed the connection)
23:57:37  * ednapiranhaquit (Remote host closed the connection)
23:59:58  * jjmalinaquit (Ping timeout: 272 seconds)