00:11:31  * paulfryzelquit (Remote host closed the connection)
00:12:06  * paulfryzeljoined
00:12:25  * mikealquit (Quit: Leaving.)
00:14:49  * kenansulaymanquit (Quit: ≈ and thus my mac took a subtle yet profound nap ≈)
00:16:36  * paulfryzelquit (Ping timeout: 252 seconds)
00:24:37  * dguttmanquit (Quit: dguttman)
00:31:22  * jcrugzz_joined
00:32:04  * jcrugzzquit (Read error: Operation timed out)
00:34:41  * jerrysvquit (Remote host closed the connection)
00:38:39  * julianduquejoined
00:42:58  * dominictarrquit (Quit: dominictarr)
00:55:06  * alexiquit (Remote host closed the connection)
00:56:01  * fallsemojoined
00:58:55  * thlorenzjoined
01:03:41  * thlorenzquit (Ping timeout: 272 seconds)
01:07:59  * mikealjoined
01:11:14  * thlorenzjoined
01:11:17  * mikealquit (Client Quit)
01:12:59  * Punnajoined
01:13:18  * Pwnnaquit (Read error: Connection reset by peer)
01:13:23  * Punnachanged nick to Pwnna
01:19:40  * fallsemoquit (Ping timeout: 240 seconds)
01:51:54  * esundahljoined
01:58:24  * dguttmanjoined
01:59:16  <levelbot>[npm] [email protected] <http://npm.im/prohibition>: Generalized geolocation management of places or things (@ednapiranha)
01:59:21  * thlorenz_joined
02:04:29  * thlorenz_quit (Ping timeout: 272 seconds)
02:07:55  * dguttmanquit (Quit: dguttman)
02:59:27  * mikealjoined
02:59:47  * thlorenz_joined
03:01:57  <chilts>if I want to put large blobs into values, can I somehow stream them or do I need the entire 'value' at the moment of the .put()?
03:02:09  <chilts>I'm happy if I do, but would like it if I didn't
03:02:33  <chilts>I've re-read over the docs but can't find anything which looks like what I'm thinking :)
03:03:30  * chiltsshould look at level-static
03:04:15  * thlorenz_quit (Ping timeout: 260 seconds)
03:04:28  <rescrv>chilts: LevelDB provides no way to stream anything
03:05:05  <rescrv>if you're storing large blobs, you can either chunk them (e.g, key<0> -> part 1, key<1> -> part2), or store pointers to large blobs on a shared resource
03:05:10  <rescrv>I'd recommend the second optin
03:06:06  <chilts>rescrv: thanks
03:06:27  <chilts>just wanted to check I was on the right track
03:06:32  <rvagg>chilts (& rescrv): already done for you here: https://github.com/juliangruber/level-stream
03:06:47  <chilts>ahah, I'll take a look at that - thanks
03:07:01  * ralphtheninjaquit (Read error: Operation timed out)
03:07:02  <chilts>I thought there might be something
03:07:46  <rvagg>of course!
03:07:54  <chilts>even though storing the blobs elsewhere are a solution I'd do in some circumstances, for this use-case I'd like them all in the DB ... just playing with some stuff really
03:07:58  <chilts>thanks rvagg
03:08:09  <rvagg>yeah, dominic is exploring something along those lines
03:08:28  <rvagg>there was a fascinating talk at lxjs by a russian guy who is storing large amounts of images and he's using a cool technique to do it efficiently
03:08:40  <rvagg>so the idea is perhaps to have a parallel store for large blobs that are referenced by level
03:09:21  <rescrv>rvagg, chilts: coolest example of this kind of thing is https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf
03:10:02  <rvagg>rescrv: yeah, this guy that spoke at lxjs has reimplemented haystack for Node
03:10:05  * rvaggdigs up links
03:10:07  <chilts>ooh, so many interesting things to read
03:10:12  * DTrejojoined
03:10:29  <chilts>I've watched a few of the lxjs talks, but missed that one : rvagg, do you remember his name?
03:11:14  <rvagg>Ivan Babrou, @ibobrik, here's his haystack clone, backpack: https://github.com/Topface/backpack
03:11:25  <chilts>oh ho ho - nice
03:11:33  <rvagg>haystack is a crazy but brilliant idea
03:15:25  * timoxleyquit (Remote host closed the connection)
03:16:00  * timoxleyjoined
03:17:19  <chilts>so it looks like it'll store each chunk at a new ('key ' + timestamp())
03:17:31  <chilts>not that I care much, just figuring out how it works
03:18:13  * julianduquequit (Ping timeout: 246 seconds)
03:20:00  <rvagg>the key is that they are preallocated slabs on disk and it just writes new data as it comes in to an empty chunk
03:20:10  <rvagg>so not individual files, big huge files with many files inside
03:20:19  * timoxleyquit (Ping timeout: 246 seconds)
03:20:20  <rvagg>fewer file descriptors to manage, simple indexes on the slabs
03:20:48  <rescrv>we're using a similar technique within our distributed filesystem
03:21:26  <chilts>kinda like allocating a lump of memory and managing it yourself, faster for a particular use-case :)
03:21:33  <rvagg>sort of
03:21:34  <chilts>cool, I'm gonna try that anyway
03:21:50  <chilts>rvagg: less resources needed I guess
03:22:03  <chilts>ah yeah, fewer file descriptors as you say
03:22:56  <rescrv>it's not just that
03:23:10  <rescrv>it's that each file has metadata, and that's often stored separately on a filesystem
03:23:35  <rescrv>there's a difference between "ls -l foo" and "cat foo"
03:23:48  <rescrv>so the slabs help, but where haystack really helped was in reigning in the metadata
03:24:38  <chilts>ah sorry, was just talking about level-stream there :)
03:24:47  <rescrv>ah, makes sense
03:24:54  <chilts>I might print off that Haystack thing to read later
03:25:21  <chilts>rescrv: so in Haystack, the meta data is in memory (for speeeed)
03:25:34  * ryan_ramagejoined
03:25:45  * julianduquejoined
03:26:10  <rescrv>chilts: I don't remember if it's 100% in memory. I think the real win was that it's colocated with the data itself, so you read that chunk, and have it all
03:26:26  <rescrv>I would assume that they'd index where those chunks are, and would put it in memory.
03:26:29  <rescrv>Facebook loves memory
03:26:31  <chilts>ok
03:26:50  <chilts>heh, it remembers *everything*
03:27:31  <rescrv>I asked them how much per machine and was told to look at the specs on the highest of high end machines, and max that out
03:27:41  <rescrv>at the time it was 256GB+ per machine
03:29:28  * mikealquit (Quit: Leaving.)
03:31:42  * jcrugzzjoined
03:32:44  * mikealjoined
03:33:02  * gwenbelljoined
03:33:47  * jcrugzz_quit (Ping timeout: 248 seconds)
03:35:28  * timoxleyjoined
03:40:27  * fallsemojoined
03:41:57  * jcrugzz_joined
03:42:40  * DTrejoquit (Remote host closed the connection)
03:43:07  * DTrejojoined
03:43:45  * jcrugzzquit (Ping timeout: 240 seconds)
03:45:27  * fallsemoquit (Ping timeout: 272 seconds)
03:46:03  * jcrugzz_changed nick to jcrugzz
03:47:25  * DTrejoquit (Ping timeout: 240 seconds)
03:59:28  * alexijoined
04:00:10  * thlorenz_joined
04:04:51  * thlorenz_quit (Ping timeout: 252 seconds)
04:06:20  * julianduquequit (Ping timeout: 260 seconds)
04:08:09  * topek_joined
04:08:15  <alexi>brycebaril rvagg: I want to apologize for the panic last night. I should've go reduced tests route before writing here. :) It was a bug in my code, it was hidden between the lines, so it was hard to spot.
04:08:27  <alexi>Thank you for spending time with me and trying to help.
04:08:52  <rvagg>alexi: heh! no probs, it wasn't a panic on my end because you were the only one reporting anything like that
04:09:05  <rvagg>I only panic if it's obviously a problem on my end or there are multiple people saying the same thing
04:09:11  <rvagg>alexi: interesting issue none the less!
04:11:47  <alexi>So I basically was overriding my records with copy of the object for less privileged accounts where some of the fields are omitted. :) It's my karma for allowing information flow freely :)
04:13:27  <rvagg>heh
04:13:45  <rvagg>so you got lev working to inspect the data then?
04:14:29  <alexi>Yep, and I inserted console.log literally every other line :)
04:15:14  <rvagg>that'll do the trick!
04:16:30  <rvagg>alexi: lev has been upgraded to the latest `level` now so it'll read .ldb files
04:16:42  * jcrugzz_joined
04:17:45  <alexi>Thank you.
04:18:11  * topek_quit (Remote host closed the connection)
04:18:55  * jcrugzzquit (Ping timeout: 260 seconds)
04:21:41  * thlorenzquit (Remote host closed the connection)
04:48:45  <levelbot>[npm] [email protected] <http://npm.im/shiro>: Online quiz game engine, inspired by famous russian tv show What? Where? When? (Million Dollar Mind Game). (@alexindigo)
04:49:11  * jcrugzzjoined
04:51:41  * jcrugzz_quit (Ping timeout: 272 seconds)
05:08:16  <levelbot>[npm] [email protected] <http://npm.im/shiro>: Online quiz game engine, inspired by russian tv show 'What? Where? When?' (Million Dollar Mind Game). (@alexindigo)
05:15:33  * thlorenzjoined
05:19:49  * gwenbellquit (Quit: Lost terminal)
05:23:26  * jxsonjoined
05:24:04  * thlorenzquit (Ping timeout: 256 seconds)
05:26:15  * jxson_quit (Ping timeout: 272 seconds)
05:28:21  * ryan_ramagequit (Quit: ryan_ramage)
05:28:25  * jxsonquit (Ping timeout: 272 seconds)
05:52:16  <levelbot>[npm] [email protected] <http://npm.im/shiro>: Online quiz game engine, inspired by russian tv show 'What? Where? When?' (Million Dollar Mind Game). (@alexindigo)
05:54:09  * ryan_ramagejoined
05:58:56  * jxsonjoined
06:01:04  * thlorenzjoined
06:05:18  * thlorenzquit (Ping timeout: 252 seconds)
06:10:05  * timoxleyquit (Remote host closed the connection)
06:10:38  * timoxleyjoined
06:15:07  * timoxleyquit (Ping timeout: 260 seconds)
06:15:46  * kenansulaymanjoined
06:26:15  * ryan_ramagequit (Quit: ryan_ramage)
06:40:55  <kenansulayman>hij1nx Could you consider publishing a lev-lmdb? We could use that
06:49:06  * kenansulaymanquit (Quit: ≈ and thus my mac took a subtle yet profound nap ≈)
06:56:36  * esundahlquit (Remote host closed the connection)
06:59:59  * timoxleyjoined
07:01:28  * thlorenzjoined
07:06:19  * thlorenzquit (Ping timeout: 272 seconds)
07:21:04  * thlorenzjoined
07:25:35  * jcrugzzquit (Ping timeout: 272 seconds)
07:25:57  * thlorenzquit (Ping timeout: 272 seconds)
07:27:41  * esundahljoined
07:36:06  * esundahlquit (Ping timeout: 256 seconds)
07:53:34  * jxsonquit (Remote host closed the connection)
08:00:19  * timoxleyquit (Remote host closed the connection)
08:00:54  * timoxleyjoined
08:01:54  * thlorenzjoined
08:05:08  * timoxleyquit (Ping timeout: 240 seconds)
08:06:39  * thlorenzquit (Ping timeout: 260 seconds)
08:16:23  * timoxleyjoined
08:21:39  * thlorenzjoined
08:25:53  * thlorenzquit (Ping timeout: 240 seconds)
08:28:13  * esundahljoined
08:31:19  * jcrugzzjoined
08:32:50  * esundahlquit (Ping timeout: 265 seconds)
08:34:46  <levelbot>[npm] [email protected] <http://npm.im/shiro>: Online quiz game engine, inspired by russian tv show 'What? Where? When?' (Million Dollar Mind Game). (@alexindigo)
08:39:25  <rvagg>lev-lmdb would certainly be interesting
09:02:16  * thlorenzjoined
09:06:55  * thlorenzquit (Ping timeout: 272 seconds)
09:59:07  * esundahljoined
10:03:47  * esundahlquit (Ping timeout: 260 seconds)
10:45:21  * jcrugzzquit (Ping timeout: 272 seconds)
10:59:35  * esundahljoined
11:03:09  * thlorenzjoined
11:03:38  * esundahlquit (Ping timeout: 240 seconds)
11:07:43  * thlorenzquit (Ping timeout: 260 seconds)
11:20:08  * insertcoffeejoined
11:23:49  <rvagg>need a description for my nodeconf.eu video, anyone around that can critique this for me before I send it in?
11:24:00  <rvagg>The term "database" has a lot of baggage but it essentially refers to a tool, external to our applications, that we use for persistence, performance and shared access. Rod introduces the new and vibrant ecosystem of Node Databases, inspired by LevelDB.
11:24:04  <rvagg>Taking a cue from Node.js philosophy, Node Databases are built around a small, extensible core of storage primitives with a rapidly growing list of extensions that provide more complex functionality.
11:24:08  <rvagg>?
11:37:40  <rescrv>rvagg: do they want it in third person? I either use first person, or just say, "This talk introduces..."
11:38:05  <rvagg>ugh, I have no idea, I'll change it and let them tinker as they see fit
11:41:06  <rescrv>other than that it it seems, having never seen you talk, that your description is good.
11:41:25  <rescrv>there'll be people who nitpick the term database, but those people probably aren't in your target audience
11:45:59  <rvagg>yeah, well I spend the first 5 minutes pulling apart the term and basically saying that it's nearly impossible to pin down to a formal definition
11:46:10  <rvagg>so I come up with my own!
11:48:10  * dominictarrjoined
12:03:33  * thlorenzjoined
12:08:06  * thlorenzquit (Ping timeout: 256 seconds)
12:27:20  * kenansulaymanjoined
12:35:06  * ralphtheninjajoined
12:37:55  * ralphtheninjaquit (Client Quit)
12:45:59  * tmcwjoined
12:46:34  * fallsemojoined
13:03:56  * thlorenzjoined
13:08:29  * thlorenzquit (Ping timeout: 272 seconds)
13:18:51  * DTrejojoined
13:20:21  * fallsemoquit (Ping timeout: 252 seconds)
13:21:52  * DTrejoquit (Remote host closed the connection)
13:22:18  * DTrejojoined
13:22:35  * ryan_ramagejoined
13:26:52  * DTrejoquit (Ping timeout: 256 seconds)
13:34:05  * ryan_ramagequit (Quit: ryan_ramage)
13:44:52  * kenansulaymanquit (Ping timeout: 264 seconds)
13:54:04  * dominictarrquit (Quit: dominictarr)
13:54:31  * kenansulaymanjoined
14:06:11  * thlorenz_joined
14:06:21  * dominictarrjoined
14:16:41  * jjmalinajoined
14:16:44  * jjmalinaquit (Max SendQ exceeded)
14:18:09  * fallsemojoined
14:20:23  * dguttmanjoined
14:23:09  * jjmalinajoined
14:23:36  * esundahljoined
14:23:43  * esundahlquit (Read error: Connection reset by peer)
14:24:06  * esundahljoined
14:34:43  * Acconutjoined
14:35:30  * Acconutquit (Client Quit)
14:35:38  * ryan_ramagejoined
14:39:14  * thlorenzjoined
14:43:47  * thlorenzquit (Ping timeout: 260 seconds)
14:58:11  * brianloveswordsquit (Excess Flood)
14:58:52  * brianloveswordsjoined
15:03:19  * mikealquit (Quit: Leaving.)
15:16:44  * ednapiranhajoined
15:30:30  * mikealjoined
15:35:31  * tmcwquit (Remote host closed the connection)
15:37:12  * Acconutjoined
15:38:06  * tmcwjoined
15:44:46  * alexiquit (Remote host closed the connection)
15:54:33  * Acconutquit (Quit: Acconut)
15:55:01  * ednapiranhaquit (Remote host closed the connection)
15:56:03  * dguttmanquit (Quit: dguttman)
16:13:48  * ednapiranhajoined
16:30:45  * dguttmanjoined
16:40:26  * thlorenzjoined
16:40:50  * mikealquit (Quit: Leaving.)
16:44:36  * thlorenzquit (Ping timeout: 245 seconds)
16:45:55  * dominictarrquit (Quit: dominictarr)
16:58:09  * Acconutjoined
16:58:36  * Acconutquit (Client Quit)
17:13:02  * jxsonjoined
17:16:01  * dominictarrjoined
17:22:39  * julianduquejoined
17:39:29  * Acconutjoined
17:43:20  * Acconutquit (Client Quit)
17:54:05  * paulfryzeljoined
18:01:29  * mikealjoined
18:11:08  * thlorenzjoined
18:16:01  * thlorenzquit (Ping timeout: 272 seconds)
18:20:43  * thlorenz_quit (Ping timeout: 248 seconds)
18:28:35  * thlorenzjoined
18:38:18  * mikealquit (Quit: Leaving.)
18:38:31  * mikealjoined
18:48:19  * timoxleyquit (Remote host closed the connection)
18:48:40  * timoxleyjoined
18:50:39  * jcrugzzjoined
18:53:31  * tmcwquit (Remote host closed the connection)
18:54:05  * tmcwjoined
18:58:31  * tmcwquit (Ping timeout: 245 seconds)
19:01:30  * tmcwjoined
19:01:47  * insertcoffeequit (Ping timeout: 248 seconds)
19:07:47  * Acconutjoined
19:08:38  * paulfryzelquit (Remote host closed the connection)
19:09:13  * paulfryzeljoined
19:13:47  * paulfryzelquit (Ping timeout: 268 seconds)
19:16:01  * Acconutquit (Quit: Acconut)
19:16:06  <ryan_ramage>question, I am thinking about the topic in #couchdb about chained map-reduce
19:16:36  <ryan_ramage>any direction on using leveldb as a chained map reduce for couch?
19:17:26  <ryan_ramage>basicially listen on the changes feed, maybe use something like https://github.com/rvagg/node-level-mapped-index
19:17:41  <ryan_ramage>and then have a second listener on that?
19:19:59  * mikealquit (Quit: Leaving.)
19:41:16  * ryan_ramage_joined
19:41:16  * ryan_ramage_quit (Changing host)
19:41:17  * ryan_ramage_joined
19:42:00  * ryan_ramagequit (Ping timeout: 260 seconds)
19:42:00  * ryan_ramage_changed nick to ryan_ramage
19:49:53  * paulfryzeljoined
19:50:29  * dominictarrquit (Quit: dominictarr)
19:59:12  * jxsonquit (Remote host closed the connection)
19:59:38  * jxsonjoined
20:00:35  <jcrugzz>ryan_ramage: what do you mean by chained map reduce? you can get live updates froma map-reduce in leveldb with https://github.com/dominictarr/map-reduce
20:00:35  <jcrugzz>and level-live-stream
20:00:55  * esundahlquit (Read error: Connection reset by peer)
20:00:58  * jxson_joined
20:01:05  * jxsonquit (Read error: Connection reset by peer)
20:01:21  * esundahljoined
20:15:40  * dominictarrjoined
20:19:36  * jcrugzzquit (Ping timeout: 245 seconds)
20:20:34  * thlorenzquit (Remote host closed the connection)
20:22:26  * Acconutjoined
20:22:34  * thlorenzjoined
20:22:36  * Acconutquit (Client Quit)
20:28:31  * davidstraussquit (Ping timeout: 245 seconds)
20:28:48  * davidstraussjoined
20:29:44  * paulfryzelquit (Remote host closed the connection)
20:30:44  * thlorenzquit (Remote host closed the connection)
20:36:26  * mikealjoined
20:41:01  * paulfryzeljoined
20:42:07  * paulfryzelquit (Read error: Connection reset by peer)
20:42:26  * paulfryzeljoined
20:47:08  * davidstraussquit (Remote host closed the connection)
20:48:41  * thlorenzjoined
20:52:13  * paulfryz_joined
20:55:26  * paulfryzelquit (Ping timeout: 240 seconds)
21:06:28  * thlorenzquit (Remote host closed the connection)
21:07:16  * thlorenzjoined
21:07:27  * thlorenzquit (Remote host closed the connection)
21:13:02  * thlorenzjoined
21:17:39  * thlorenzquit (Ping timeout: 259 seconds)
21:18:05  * tmcwquit (Remote host closed the connection)
21:23:28  * mikealquit (Quit: Leaving.)
21:28:37  * jxson_quit (Remote host closed the connection)
21:29:03  * jxsonjoined
21:33:26  * jxsonquit (Ping timeout: 240 seconds)
21:33:54  * julianduquequit (Quit: leaving)
21:37:57  * thlorenzjoined
21:38:27  * mikealjoined
21:46:11  * ryan_ramagequit (Quit: ryan_ramage)
21:46:39  * thlorenzquit (Ping timeout: 272 seconds)
21:53:04  * ryan_ramagejoined
21:55:15  * jcrugzzjoined
21:57:43  * thlorenzjoined
22:02:45  * Pwnnaquit (Ping timeout: 272 seconds)
22:05:09  * Pwnnajoined
22:06:38  * thlorenzquit (Remote host closed the connection)
22:12:51  * thlorenzjoined
22:23:47  * paulfryz_quit (Remote host closed the connection)
22:24:11  * thlorenzquit (Remote host closed the connection)
22:24:24  * paulfryzeljoined
22:27:26  * jcrugzzquit (Ping timeout: 240 seconds)
22:28:38  * paulfryzelquit (Ping timeout: 240 seconds)
22:28:51  * ryan_ramagequit (Quit: ryan_ramage)
22:34:01  * jcrugzzjoined
22:45:07  * brycebarilquit (Quit: Leaving.)
22:47:52  * jxsonjoined
23:05:54  * mikealquit (Quit: Leaving.)
23:13:06  * jcrugzzquit (Ping timeout: 245 seconds)
23:16:38  * julianduquejoined
23:20:45  * jjmalinaquit (Quit: Leaving.)
23:24:07  * esundahlquit (Remote host closed the connection)
23:24:33  * esundahljoined
23:25:43  * paulfryzeljoined
23:25:47  * paulfryzelquit (Remote host closed the connection)
23:28:38  * esundahlquit (Ping timeout: 240 seconds)
23:30:20  * ralphtheninjajoined
23:30:49  * ralphtheninjaquit (Client Quit)
23:35:52  * fallsemoquit (Quit: Leaving.)
23:53:31  * ednapiranhaquit (Remote host closed the connection)
23:56:16  * julianduquequit (Ping timeout: 245 seconds)