00:01:35  <chilts>yeah, I've been trying to think how I can do DynamoDBs atomic stuff in MoDB but have decided I don't know how to do it and I don't care enough to try :)
00:01:47  <chilts>I think people find ways around limitations anyway, unless it's toooo limited
00:02:02  <chilts>so people write code dependant on what their datastore properties are
00:12:54  <rvagg>monolithic databases are sadly like dictators, forcing us to adopt a particular architecture when their only real job is to HOLD ON TO STUFF FOR US
00:59:35  * ramitosquit (Quit: Textual IRC Client: www.textualapp.com)
01:26:28  * werlejoined
01:29:35  * timoxleyjoined
01:38:30  * thl0joined
01:50:22  <st_luke>even worse they're too valuable to most companies to do things in the interest of users
01:55:17  <rvagg>imagine if we purchased and used filesystems like we do databases
01:55:20  <rvagg>crazyness
01:58:20  <thl0>so range queries only work on keys and if I wanna query on other values I gotta index that so it becomes a key as well?
01:58:29  <st_luke>yeah, fortunately filesystem as a service hasn't caught on yet
01:58:46  * timoxleyquit (Quit: Textual IRC Client: www.textualapp.com)
01:58:53  <st_luke>thl0: get a little more descriptive in your keys
01:59:01  <thl0>?
01:59:14  <thl0>talking about namespacing?
01:59:33  <thl0>ah so you mean the keys should hint at the values?
01:59:52  <st_luke>yeah put a critical bit of the value in the key so you know what you're looking for
02:00:02  <thl0>ah - something clicked just now
02:00:10  <thl0>so keys are kinda like precanned queries
02:00:41  * timoxleyjoined
02:00:57  <thl0>starting to get it I think
02:01:33  <st_luke>rangequeries are super valuable
02:01:39  <st_luke>and I don't mean they're expensive
02:02:20  <thl0>understood since it's all presorted and only hits the smallest possible layer first right?
02:02:51  <thl0>rangequeries are the only queries supported by leveldb right?
02:03:06  <st_luke>get
02:04:40  <rvagg>we really need to put together some good documentation on how to construct your keys
02:04:49  <thl0>st_luke: but get requires exact key right?
02:04:51  <rvagg>perhaps we can talk dominictarr into starting something off
02:05:19  <rvagg>thl0: see https://github.com/rvagg/node-levelup/wiki/Modules there's a few extensions that do indexing-type things
02:05:33  <thl0>rvagg: here is the deal if you guys keep helping me to get started quickly with this I may write a short tutorial on this as a blog post ;)
02:06:03  <rvagg>thl0: see https://github.com/dominictarr/level-index and https://github.com/rvagg/node-level-mapped-index for a couple of indexing implementations
02:06:24  <thl0>rvagg: saw these modules, but want to avoid adding more layers until I understand at least levelup (which is already at least 2 layers up)
02:06:30  <rvagg>thl0: the latter gives you a db.getBy() method that can fetch by index name
02:06:32  <rvagg>yeah, fair enough
02:06:32  <st_luke>there's a tutorial in levelup/README.md
02:07:15  <thl0>st_luke: doesn't go into queries very much (unless I missed something)
02:07:48  <st_luke>you read this right - https://github.com/rvagg/node-levelup#dbcreatereadstreamoptions
02:08:08  <st_luke>rvagg: do you get an irc notification every time someone links to one of your repos, since it's the same username
02:08:28  <thl0>rvagg: will read through these modules though tomorrow at work
02:08:47  <rvagg>st_luke: yeah, I set one up for 'leveldb' when I first did levelup, these days it's mentioned a bit too frequently!
02:09:27  <thl0>st_luke: I skimmed through it, but it didn't fully explain how these range queries actually work (i.e. that all the magic is in the keys)
02:10:40  <st_luke>'LevelDB stores entries sorted lexicographically by keys.'
02:10:41  <st_luke>that part :) ?
02:10:47  <thl0>now where you guys pointed me in the right direction, I should be able to figure this out, thanks
02:11:27  <thl0>st_luke: bear with me - my first time using any key value store, so some concepts are not obvious right away ;)
02:11:34  <st_luke>ah I see
02:11:40  <st_luke>I thought you were a little familiar with some other ones
02:11:59  <thl0>but as I said things start clicking ;)
02:12:14  <thl0>only worked with sql and nosql (object dbs) so far
02:12:31  <st_luke>I guess it might not be as quickly apparent why rangequeries in leveldb are a really incredible way to retrieve data when you compare queries in other k/v stores.
02:13:02  <thl0>starting to get there for me though
02:13:23  <rvagg>it's all about the keys! you have to completely adjust your thinking to the ordered nature of leveldb and how you can take maximum advantage of it with your keys
02:13:43  <thl0>rvagg: starting to see that - is there an example somewhere with keys well done?
02:13:55  <st_luke>yeah, once you start to grok how that works it opens up a lot
02:14:01  <rvagg>thl0: and you could consider level-sublevel as an important part of the puzzle, it's another layer but it's an important layer and can help with organising things into the equivalent of "tables"
02:14:38  <thl0>rvagg: https://github.com/dominictarr/level-sublevel ?
02:14:59  <thl0>cool more code to read tomorrow ;)
02:15:29  <rvagg>yar
02:16:24  <thl0>so when I get this I'll try to write a tutorial especially at people like me who come from sql/nosql understanding
02:16:47  <rvagg>excellent
02:16:55  <rvagg>stick it on https://github.com/rvagg/node-levelup/wiki/Resources when you have something
02:18:44  <thl0>rvagg: will do - just give me a few days with this
02:25:54  <thl0>one thing that's confusing is how this example makes sense: https://github.com/rvagg/node-levelup#dbcreatewritestreamoptions
02:26:05  <thl0>i.e keys are 'name', 'dob', etc.
02:26:25  <thl0>doesn't look helpful to range queries
02:26:25  <rvagg>mm
02:26:29  <rvagg>no, it's not!
02:26:34  <rvagg>it's just a naive example
02:27:07  <rvagg>I wish I had a good public example to point to but I can't think of one
02:27:21  <thl0>got it, just confusing since it makes you think it works kinda like an object database
02:27:46  <thl0>rvagg: no problem for me anymore since you guys set me straight, just wanted to point it out
02:28:19  <rvagg>thl0: some interesting stuff to look at for helping with coming up with good sortable keys: https://github.com/dominictarr/monotonic-timestamp https://github.com/chilts/flake https://github.com/deanlandolt/bytewise
02:29:22  <thl0>rvagg: thanks, bookmarked
02:29:52  <chapel>rvagg: also unid that I created
02:30:07  <thl0>chapel: link?
02:30:13  <rvagg>we need a section for these on the modules page if we don't already
02:31:11  <thl0>chapel: never mind, found it
02:31:43  <chapel>Lol was taking too long to type on my phone
02:31:56  <chapel>No readme sorry
02:32:14  <thl0>I need to read code anyways, so no problem
02:32:42  <chapel>I moved to the bay area and am in a hotel. Haven't had time to code other than work
02:33:06  <thl0>so writeStream and batch basically serve same purpose right?
02:34:31  <rvagg>thl0: sort of, a WriteStream does batch() within a node tick, so mostly you'll be batching single elements unless they come in at the same time
02:34:44  <rvagg>and a WriteStream can be kept around for as long as you like, you can use it as an easy write-mechanism if you like
02:34:59  <rvagg>it's mainly a convenience really, and a mirror to ReadStream
02:35:19  <thl0>ok, so unless I have a good reason batch is the way to go then?
02:35:24  <rvagg>probably
02:35:58  <thl0>btw if I have multiple batches going on at the same time, how would I know when to close the db without doing the count-- thing?
02:36:22  <rvagg>at the moment you need to keep track for yourself
02:36:44  <thl0>ok, but is it expected that for an app I'll keep the connection open for the app's lifetime?
02:36:47  <rvagg>at the top of my TODO list is an auto-cleanup-on-close thing but right now it's a danger point, keep track or you might get in big trouble
02:36:52  <rvagg>yep
02:36:55  <thl0>ok
02:40:31  <thl0>rvagg: so the batch chained form returns a deferred - but how does it know when to start executing from the top?
02:40:45  <thl0>i.e. I could do var b = batch()
02:40:52  <rvagg>thl0: it's not a deffered, it's a collection
02:40:57  <rvagg>like an array
02:41:14  <rvagg>b.put('foo', 'bar') -- think of it like b.push({ key: 'foo', 'bar' })
02:41:21  <rvagg>until you call b.write() nothing will happen, you can discard it and it'll go nowhere
02:41:38  <thl0>rvagg: ok so write
02:41:53  <thl0>what I was thinking, wasn't obvious
02:42:17  <rvagg>thl0: btw, if any of this is unclear from the docs then please submit a PR; when you have your head too deeply in something it's difficult to see it from the outside
02:42:54  <thl0>rvagg: sure, I'll try to remember what was not obvious and once I have a good understanding I'll try to help improve things
02:55:12  <hij1nx>juliangruber: hey, what's up with this not being used? https://github.com/juliangruber/multilevel/blob/master/lib/client.js#L7
02:57:52  * brianloveswordsquit (Excess Flood)
02:59:17  * brianloveswordsjoined
03:01:14  <thl0>rvagg: gotta go, but it seems like db.batc().put() .. is broken unless you use the callback way of opening db
03:01:25  <thl0>I'll post a gist tomorrow
03:01:32  <rvagg>thl0: ahh, yeah... that could be right
03:01:34  <rvagg>darn it
03:02:03  <thl0>no worries - I'll file an issue
03:03:13  * thl0quit (Remote host closed the connection)
03:07:15  <rvagg>tbh I'm not a big deferred-open user, I prefer the surety of levelup('path', function (err, db) { /* use `db` here */ })
03:07:27  <rvagg>but we have a test suite for it and the chained batch form is obviously missing from there since it's so new
03:09:59  * werlequit (Ping timeout: 260 seconds)
03:11:07  * werlejoined
03:45:22  * werlequit (Quit: Leaving.)
04:22:40  * Pwnnajoined
04:49:52  * timoxleyquit (Quit: Computer has gone to sleep.)
05:21:57  * timoxleyjoined
05:27:06  * Pwnnaquit (Quit: mrrow~)
05:27:17  * Pwnnajoined
05:31:30  * Punnajoined
05:36:27  * Punnaquit (Ping timeout: 256 seconds)
05:38:09  * st_lukequit (Remote host closed the connection)
05:40:27  * Punnajoined
05:44:24  * Punnaquit (Client Quit)
05:50:03  * dominictarrquit (Quit: dominictarr)
06:42:52  * timoxleyquit (Quit: Computer has gone to sleep.)
06:59:36  * timoxleyjoined
07:06:07  <juliangruber>hij1nx: that was before dominictarr's extensive rewrite, need to throw that out
07:07:13  * Pwnnaquit (Ping timeout: 276 seconds)
07:15:01  * werlejoined
07:25:37  * timoxleyquit (Quit: Computer has gone to sleep.)
07:38:29  * werlequit (Quit: Leaving.)
07:45:27  * timoxleyjoined
07:48:55  * dominictarrjoined
07:50:16  * dominictarrquit (Read error: Connection reset by peer)
07:53:54  * dominictarrjoined
08:26:38  * mcollinajoined
08:41:37  * ChrisPartridgequit (Ping timeout: 256 seconds)
08:46:42  <juliangruber>threw it on the ground!
09:03:29  * dominictarrquit (Quit: dominictarr)
09:25:45  * dominictarrjoined
09:50:03  * dominictarrquit (Quit: dominictarr)
10:26:40  * timoxleyquit (Quit: Computer has gone to sleep.)
10:39:07  * timoxleyjoined
10:40:34  * ChrisPartridgejoined
11:33:13  * rvaggquit (Quit: ta ta)
11:36:12  * rvaggjoined
12:12:21  * thl0joined
12:13:47  <thl0>using levelup: when I use valueEncoding 'json' and put some things and call db.createReadStream it fails with 'unexpected token "w"'
12:13:56  <thl0>db.get works however
12:14:23  <thl0>is that an issue or is it me doing things wrong?
12:15:52  <thl0>I can use default encoding and readStream works, but then I'd have to manually stringify and parse
12:16:58  <thl0>rvagg: juliangruber: hij1nx: should I file an issue?
12:21:27  <juliangruber>thl0: i think you can do db.createReadStream({valueEncoding:'json'})
12:21:41  <thl0>tried that ;)
12:21:44  <juliangruber>or did you specify the valueEncoding on the db object?
12:21:45  <thl0>no worky
12:21:47  <juliangruber>hmm
12:21:51  <thl0>yes both
12:21:51  <juliangruber>can you create a gist?
12:21:54  <thl0>ok
12:22:03  <thl0>will make an issue real quick
12:29:06  <thl0>juliangruber: I think I figured out what the problem was
12:29:19  <juliangruber>thl0: cool, what was it?
12:29:20  <thl0>I had written to same db previously with default encoding
12:29:29  <juliangruber>aah
12:29:35  <thl0>and those entries where still there - tricky ;)
12:30:32  <thl0>ok I'm starting to jot down these points, so I can make a nice post for others who start using level
12:40:09  * thl0quit (Remote host closed the connection)
12:52:38  * timoxleyquit (Quit: Computer has gone to sleep.)
13:09:48  * werlejoined
13:17:05  * timoxleyjoined
13:23:48  * timoxleyquit (Quit: Computer has gone to sleep.)
13:25:32  * werlequit (Ping timeout: 260 seconds)
13:28:44  * levelbotjoined
13:34:57  * werlejoined
13:51:04  * thl0joined
14:28:04  * timoxleyjoined
14:51:33  * timoxleyquit (Quit: Computer has gone to sleep.)
15:13:17  * timoxleyjoined
15:35:17  * ChrisPartridgequit (Ping timeout: 256 seconds)
15:49:41  * timoxleyquit (Quit: Computer has gone to sleep.)
16:00:03  * Pwnnajoined
16:05:19  * thl0quit (Ping timeout: 252 seconds)
16:15:53  * thl0joined
16:34:52  * thl0quit (Remote host closed the connection)
16:37:29  * brianloveswordsquit (Excess Flood)
16:38:49  * brianloveswordsjoined
17:20:16  * st_lukejoined
18:10:21  * thl0joined
18:14:39  * thl0quit (Ping timeout: 245 seconds)
18:23:41  * Pwnnaquit (Remote host closed the connection)
18:31:56  * Pwnnajoined
18:32:59  * brianloveswordsquit (Excess Flood)
18:34:49  * brianloveswordsjoined
18:52:01  * thl0joined
18:57:00  * ramitosjoined
19:07:20  * thl0quit (Remote host closed the connection)
19:10:50  * thl0joined
19:11:47  * st_lukequit (Read error: Connection reset by peer)
19:12:15  * st_lukejoined
19:16:14  * thl0quit (Remote host closed the connection)
19:42:21  * thl0joined
20:13:00  * st_lukequit (Remote host closed the connection)
21:05:05  * levelbotquit (Remote host closed the connection)
21:05:34  * levelbotjoined
22:07:51  * ramitosquit (Quit: Textual IRC Client: www.textualapp.com)
22:10:37  * thl0quit (Remote host closed the connection)
22:55:29  * brianloveswordsquit (Excess Flood)
22:57:51  * brianloveswords_joined
22:57:52  * ChrisPartridgejoined
22:57:53  * brianloveswords_changed nick to brianloveswords
23:40:14  * CPartridgejoined
23:47:44  * werlequit (Quit: Leaving.)
23:57:38  * st_lukejoined