00:00:01  * ircretaryquit (Remote host closed the connection)
00:00:08  * ircretaryjoined
00:03:50  * mikolalysenkojoined
00:12:32  * mikolalysenkoquit (Ping timeout: 256 seconds)
00:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 30]
00:31:01  * shuaibquit (Quit: Computer has gone to sleep.)
00:33:38  * yorickquit (Remote host closed the connection)
01:00:59  * st_lukejoined
01:05:20  * thl0joined
01:14:18  * fallsemoquit (Quit: Leaving.)
01:16:34  * jxson_joined
01:17:43  * jxson_quit (Remote host closed the connection)
01:17:50  * jxson_joined
01:18:32  * mikolalysenkojoined
01:19:35  * jxsonquit (Ping timeout: 252 seconds)
01:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 40]
01:23:17  * mikolalysenkoquit (Ping timeout: 248 seconds)
01:36:41  * ralphtheninjaquit (Ping timeout: 245 seconds)
01:44:01  * jcrugzzquit (Ping timeout: 246 seconds)
01:47:06  * jxson_quit (Remote host closed the connection)
01:57:17  * defunctzombie_zzchanged nick to defunctzombie
02:07:11  * defunctzombiechanged nick to defunctzombie_zz
02:12:24  * crankquit (Ping timeout: 260 seconds)
02:19:04  * shamaquit (Remote host closed the connection)
02:21:42  <rowbit>Daily usage stats: [developer: 3, free: 662]
02:21:43  <rowbit>Hourly usage stats: [developer: 0, free: 26]
02:46:23  * crankjoined
02:57:33  * jxsonjoined
03:02:59  * jxsonquit (Ping timeout: 252 seconds)
03:08:09  * thl0quit (Remote host closed the connection)
03:09:33  * tilgovijoined
03:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 27]
03:45:26  * tmcwjoined
03:50:36  * jcrugzzjoined
04:20:35  * st_lukequit (Remote host closed the connection)
04:20:47  * st_lukejoined
04:21:42  <rowbit>Hourly usage stats: [developer: 1, free: 3]
05:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 9]
05:58:32  * st_lukequit (Remote host closed the connection)
06:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 10]
06:22:38  * jxsonjoined
06:23:36  * jxsonquit (Read error: Connection reset by peer)
06:24:00  * jxsonjoined
06:52:09  * jxsonquit (Remote host closed the connection)
07:06:48  * CoverSlidequit (Ping timeout: 264 seconds)
07:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 13]
07:46:10  * saijanai_joined
08:03:00  * devaholicquit (Ping timeout: 276 seconds)
08:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 31]
08:37:18  * harrowquit (Quit: ERC Version 5.3 (IRC client for Emacs))
08:45:36  <juliangruber>dominictarr: holy! :) I'd propose 2 changes: 1) no db.sublevels['foo'], just db.sublevel('foo') 2) add a method with one function: foo.set('inc', {type: 'async'}, function (f, cb) {})
08:46:53  <dominictarr>oh, yeah, I was gonna add the sublevel function
08:48:59  <dominictarr>not sure about the set('inc', {…}) because I want each sublevel plugin to declare the api it adds, so you can plugin a bunch of stuff and it just works with the client
08:49:47  <juliangruber>oh I see
08:49:53  <juliangruber>then that's better
08:56:25  * jibayjoined
08:58:15  <jesusabdullah>\o/
09:02:22  * harrowjoined
09:08:54  <juliangruber>dominictarr: https://github.com/substack/undirender/pull/1 might be useful for you too
09:09:55  <dominictarr>oh, that is nice
09:11:54  * jcrugzzquit (Ping timeout: 276 seconds)
09:13:08  <dominictarr>juliangruber: what are you refering to in this comment https://github.com/juliangruber/multilevel/pull/8#commitcomment-3007629
09:13:16  <dominictarr>"hides mdm's methods?"
09:14:59  <dominictarr>juliangruber: just had a brain wave - okay so to do two way mux-demuxes, with createStream at each end, you just have to support a request header, and a response header
09:15:47  <dominictarr>currently, mux-demux only supports a request header - so only the first one to create the stream gets to send metadata
09:16:20  <dominictarr>a more symmetrical protocol would be better.
09:16:49  <rvagg>dominictarr: more and more activity happening in ##leveldb, you may want to think about lurking some time
09:16:59  <dominictarr>substack: did you see this https://twitter.com/dominictarr/status/323143464516722690
09:17:53  <dominictarr>rvagg: okay, I've set #leveldb to autojoin
09:18:11  <rvagg>try the ## version
09:18:42  <dominictarr>why two?
09:18:54  <rvagg>someone else was using #leveldb when we started
09:19:13  <rvagg>and these days freenode asks people to register 'non official' channels as ##
09:19:13  <dominictarr>some other leveldb people?
09:19:15  <rvagg>stupid
09:19:28  <dominictarr>oh, okay
09:19:50  <rvagg>yeah, since I've been in #leveldb though nobody other than node folk has really said anything
09:20:53  <dominictarr>you need at least 30 people in an irc room before any action really starts,
09:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 53]
09:31:33  * nicholas_quit (Read error: Connection reset by peer)
09:31:39  * nicholas_joined
09:40:43  * dominictarrquit (Quit: dominictarr)
09:41:09  * jcrugzzjoined
09:46:13  * ralphtheninjajoined
09:46:13  * jcrugzzquit (Ping timeout: 248 seconds)
10:04:16  * dominictarrjoined
10:21:42  <rowbit>Hourly usage stats: [developer: 2, free: 15]
11:21:42  <rowbit>Hourly usage stats: [developer: 4, free: 9]
11:36:12  * nicholas_quit (Ping timeout: 276 seconds)
11:43:36  * tmcw_joined
11:43:37  * tmcwquit (Read error: Connection reset by peer)
11:45:27  * tilgoviquit (Remote host closed the connection)
11:59:10  * devaholicjoined
12:09:09  <rowbit>/!\ ATTENTION: (default-local) [email protected] successfully signed up for developer browserling plan ($20). Cash money! /!\
12:09:09  <rowbit>/!\ ATTENTION: (default-local) paid account successfully upgraded /!\
12:21:42  <rowbit>Hourly usage stats: [developer: 1, free: 15]
12:24:12  * yorickjoined
12:27:48  * tmcw_quit (Remote host closed the connection)
12:28:29  * jlordquit (Remote host closed the connection)
12:28:30  * mbalhoquit (Remote host closed the connection)
12:30:31  * tmcwjoined
12:34:18  * ITproquit (Ping timeout: 256 seconds)
12:35:31  * tmcwquit (Remote host closed the connection)
12:36:07  * ITprojoined
12:43:42  * thl0joined
12:47:57  * tmcwjoined
12:56:11  * tmcwquit (Remote host closed the connection)
12:57:12  * tmcwjoined
12:57:41  * ITproquit (Ping timeout: 248 seconds)
13:07:11  * jcrugzzjoined
13:10:38  * thl0quit (Remote host closed the connection)
13:11:41  * jcrugzzquit (Ping timeout: 245 seconds)
13:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 31]
13:25:13  * saijanai_quit (Quit: saijanai_)
13:30:46  * thl0joined
13:37:53  * st_lukejoined
13:37:54  * st_lukequit (Remote host closed the connection)
13:49:29  * ins0mniajoined
13:52:57  * dominictarrquit (Quit: dominictarr)
14:16:35  * defunctzombie_zzchanged nick to defunctzombie
14:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 25]
14:45:51  * yorickquit (Remote host closed the connection)
14:48:49  * fallsemojoined
15:05:54  <juliangruber>dominictarr: by doing `var db = duplex(mdm, mdm)` you can't call `db.createStream('meta')` anymore. you hide those mux-demux methods. hm, that probably causes overhead
15:11:48  * tmcwquit (Remote host closed the connection)
15:12:24  * tmcwjoined
15:12:33  * tmcwquit (Read error: Connection reset by peer)
15:13:04  * tmcwjoined
15:18:29  * tmcwquit (Remote host closed the connection)
15:21:29  * tmcwjoined
15:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 21]
15:24:48  * fallsemoquit (Quit: Leaving.)
15:31:37  * shuaibjoined
15:44:23  * jcrugzzjoined
16:04:59  * mbalhojoined
16:06:11  * shuaibquit (Ping timeout: 252 seconds)
16:06:13  * jlordjoined
16:13:24  * ins0mniaquit (Ping timeout: 264 seconds)
16:18:32  * ITprojoined
16:21:31  * fallsemojoined
16:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 22]
16:37:24  * shuaibjoined
16:37:36  * fallsemoquit (Quit: Leaving.)
17:07:23  * dominictarrjoined
17:09:19  * ins0mniajoined
17:12:31  * devaholicquit (Ping timeout: 245 seconds)
17:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 27]
17:55:13  * hij1nxquit (Ping timeout: 240 seconds)
18:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 26]
18:22:24  * shuaibquit (Quit: Computer has gone to sleep.)
18:32:16  * calvinfoquit (Ping timeout: 258 seconds)
18:35:46  * ITproquit (Ping timeout: 256 seconds)
18:42:46  <thl0>chrisdickinson: juliangruber: check it out: http://thlorenz.github.io/scriptie-talkie-embed/
18:44:24  * calvinfojoined
19:01:24  <dominictarr>Raynos: ping
19:18:56  * shuaibjoined
19:21:37  * shuaibquit (Client Quit)
19:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 14]
19:23:47  * yorickjoined
19:24:12  * ITprojoined
19:25:18  <dominictarr>ins0mnia: you there?
19:26:30  <ins0mnia>dominictarr: yeah (alert popped up on my screen)
19:26:44  <ins0mnia>what's up
19:27:06  <dominictarr>just working on some leveldb stuff
19:27:23  <dominictarr>it's really all beginning to come together here
19:27:50  <dominictarr>I built this 'ladder' yesterday
19:27:52  <dominictarr>https://github.com/juliangruber/multilevel/pull/8
19:28:06  <ins0mnia>dominictarr: cool I'm looking forward to start looking into level
19:28:10  <ins0mnia>level*
19:28:19  <dominictarr>It will provide us access to a whole new layer of low-hanging-fruit
19:29:14  <ins0mnia>ohh this is so cool
19:29:21  <ins0mnia>level all the way to the browser!!!
19:29:35  <dominictarr>yes, with all the plugins, etc
19:30:08  <dominictarr>needs a few more bits, but they should pretty much just work, if they use streams, or async callbacks
19:30:23  <ins0mnia>I'm thinking level+muxdemux+observable+hyperscript
19:30:26  <ins0mnia>awesome combo
19:30:43  <dominictarr>yeah, thats what I'm thinking too
19:31:12  <dominictarr>you just write one file that defines the data you have, what queries there are, etc
19:31:51  <dominictarr>then you just drop in multilevel, and you can access it from the browser
19:32:08  <dominictarr>so, you just have to define how to view that
19:32:20  <dominictarr>and you have a dynamic realtime app
19:32:34  <ins0mnia>cool
19:34:35  <jez0990>level persistance is next on my agenda after this crdt module, I'm looking forward to it
19:34:42  <jez0990>dominictarr: regarding "check if any currently existing items should be in the set. currently, one must create the set before recieving anything."
19:35:06  <dominictarr>jez0990: if that is based on crdt then it will work with level-scuttlebutt already!
19:35:27  <jez0990>if you haven't touched that since I will implement it now...?
19:36:20  <jez0990>I've been in split-minds as to whether to start with a clean slate on this module, but I'm nearly done now :D
19:37:26  <dominictarr>jez0990: if you havn't messed with the underling protocol, it will work.
19:38:45  <jez0990>then that's awesome!! about the questions above however, it was a TODO you'd written on line 41 in set.js
19:38:52  <jez0990>*question
19:40:11  <dominictarr>oh, it means that you have to create the set in the first tick
19:41:05  <dominictarr>it's not really that big a deal to fix, you'd just have to iterate over all the rows and check if they should be in the set.
19:42:31  * thl0quit (Remote host closed the connection)
19:47:05  * ecquit (Ping timeout: 245 seconds)
19:48:14  <jez0990>dominictarr: hmm, so I'm confused
19:48:51  <jez0990>if that behaviour isn't implemented how does "test - create set later" in test/sets.js work?
19:50:25  <dominictarr>you are right - that comment must be out of date
19:50:43  <dominictarr>I'm not sure where the code that implements that is
19:50:48  <dominictarr>maybe in doc?
19:51:40  <dominictarr>maybe this https://github.com/dominictarr/crdt/blob/master/set.js#L88
19:52:24  <dominictarr>hmm… but that is just an event emitter
19:53:00  <jez0990>:) even if it's magic, I'm happy
19:53:20  * hij1nxjoined
19:53:54  <dominictarr>there must be something somewhere
19:55:47  <jez0990>I am trying to find it so I can work out what actually happens with "create seq later" (which doesn't exist as a test)
19:57:37  <jez0990>oh wait, although I was trying to find it to work out what happens with Seq
19:57:41  <jez0990>oops
19:57:55  <jez0990>meant to say "oh wait, https://github.com/dominictarr/crdt/blob/master/set.js#L111"
19:59:00  <dominictarr>seq is just set with some extra methods
19:59:20  <dominictarr>there we go
20:00:28  <dominictarr>okay, just patched crdt to remove that comment
20:00:33  <dominictarr>thanks
20:02:27  * shuaibjoined
20:08:22  <jez0990>np, I think I actually understand this all now... except how the Set's on('change') is guaranteed to be called before the inheriting Seq's on('change')
20:08:44  <jez0990>is it because eventemitters form a queue? or is it not guaranteed?
20:08:55  <jez0990>dominictarr: ^
20:10:07  <dominictarr>yes, it is important to understand the order listeners are registered
20:10:30  <dominictarr>the first listener registered is the first listener triggered
20:11:33  <dominictarr>seq calls set before it adds any of it's own stuff https://github.com/dominictarr/crdt/blob/master/seq.js#L31
20:13:10  <jez0990>interesting, thanks
20:15:04  <Raynos>dominictarr: pong
20:16:09  <dominictarr>Raynos: just been thinking about level-* and the possibility of setting up your data as a series of declarative relationships
20:16:26  * ITproquit (Ping timeout: 252 seconds)
20:16:29  <dominictarr>pipelines, between different types of data
20:16:57  * ITprojoined
20:17:05  * ITproquit (Client Quit)
20:17:32  * ITprojoined
20:18:01  <Raynos>that sounds like declarative map reduce
20:18:09  <Raynos>your saying "this segment of the keyspace is RAW data"
20:18:19  <Raynos>"this segment of the keyspace is a declarative view defined by relationship X"
20:18:29  <Raynos>"this segment of the keyspace is a computed view defined by map reduce Y"
20:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 23]
20:24:29  * sandfoxjoined
20:24:31  * shuaibquit (Quit: Computer has gone to sleep.)
20:28:04  * spionjoined
20:30:44  * fallsemojoined
20:32:09  <Raynos>dominictarr: I think this idea is a very good idea.
20:32:32  <Raynos>I currently have a problem with mongoDB where setting up relationships & pipelines & transformations from one collection to another is hard, tedious & manual
20:34:03  <dominictarr>how do you even do that in mongo?
20:34:25  * thl0joined
20:35:04  <dominictarr>Raynos: and also, this segment is transformed from that segment, (on another db on another server)
20:36:21  <Raynos>well in mongo
20:48:18  <dominictarr>Raynos: I'm gessing you in mongo you just save it in each place whenever you save it
20:48:29  <Raynos>oh wait
20:48:41  <Raynos>In mongo I set up a mapReduce query from collection A to collection B
20:48:47  <Raynos>then I run that mapReduce query in a cron job every hour
20:49:06  <Raynos>so "segment of keyspace B is a computed view defined by map reduce Y but is stale for an hour"
20:49:55  <dominictarr>right
20:50:23  <dominictarr>so that is acceptable for some things...
20:50:45  <dominictarr>but not for information like whether your pizza is ready
20:51:19  <dominictarr>I'd want to know that within seconds
20:51:41  * st_lukejoined
20:51:46  <dominictarr>hmm, that is a good example for soft-realtime
20:56:12  <Raynos>well actually
20:56:18  <Raynos>I use this map reduce for a real time system
20:56:21  <Raynos>and run it every minute
20:56:56  <Raynos>and I have a system setup where you can merge a subset of "segment of keyspace B" with changes made to "segment of keyspace A" on the fly in real time
20:57:12  <dominictarr>A minute is a bit on the slow side for pizza, I'd be getting annoyed if my pizza was sitting there a whole minute and you hadn't done anything
20:57:13  <Raynos>keyspace B is really just for efficient querying of initial state
20:57:22  <Raynos>keyspace A is for real time stuff
20:57:43  <Raynos>it's basically the lambda architecture from that blog post
20:57:51  <Raynos>i actually want to rewrite it to be leveldb based
20:58:20  <dominictarr>yeah, we need more query plugins
21:13:54  * tmcw_joined
21:13:54  * tmcwquit (Read error: Connection reset by peer)
21:14:55  <mbalho>+1
21:21:42  <rowbit>Hourly usage stats: [developer: 0, free: 26]
21:28:23  * ITproquit (Read error: Connection reset by peer)
21:31:58  * sandfoxquit (Quit: sandfox)
21:33:08  * st_lukequit (Remote host closed the connection)
21:42:08  * st_lukejoined
21:42:12  <thl0>hey Raynos didn't you work on interactive blog post code samples at some point?
21:43:17  * nicholas_joined
21:45:18  <dominictarr>so hij1nx if you use level-fix-range then it will convert min/max to start/end
21:45:51  <dominictarr>but you'll still get the next item at the end - and an error if the max is after the last item in the db
21:45:57  <Raynos>thl0: http://raynos.graphics-server.jit.su/full/mario.js
21:46:01  <Raynos>that's the closst I go
21:46:20  <Raynos>dominictarr: I feel like I want to use js based map reduce and figure out how to require high order functions for it
21:46:46  * notalexgordon_joined
21:46:56  <dominictarr>I've patched this in level-sublevel, but we are still discussing the ins and outs of merging in into level{up,down}
21:47:13  <dominictarr>personally, I'm leaning towards a fix in leveldown being best
21:47:37  <dominictarr>Raynos: what do you mean?
21:48:22  <thl0>Raynos: that's pretty cool - lots of info though
21:48:34  <Raynos>dominictarr: https://github.com/Colingo/snapshot-reduce/blob/master/index.js this map reduce query is 300 lines of unmodularized code
21:48:35  <dominictarr>(have also revised my previous opinion that map-reduce was so great, there are more ways to do it)
21:48:48  <Raynos>I think if I had the ability to modularize and require functions I could do cool stuff
21:48:52  <dominictarr>oh no
21:48:53  <thl0>Raynos: totally different approach than waht I came up with: http://thlorenz.github.io/scriptie-talkie-embed/
21:49:13  <dominictarr>Raynos: well, since levelup is in your app you can use npm
21:49:25  <dominictarr>you don't have to send the map-reduce code to a server
21:51:29  <dominictarr>Raynos: I'm kinda wondering if there is a way to just pipe from one db to another…
21:51:48  <dominictarr>like master-slave replication with a transform.
21:53:00  <substack>dominictarr: you should make a scuttlebutt/rumours website with basic instructions about how all the pieces fit together and how to get started
21:53:29  <dominictarr>with screencasts
21:53:40  <thl0>Raynos: how does it know to render the mariostate, in all other examples you had a render(main, ..)?
21:53:42  <substack>yep!
21:53:44  <substack>voxeljs.com style
21:54:21  <Raynos>thl0: it has the render call
21:54:43  <dominictarr>substack: yes, I'm seeing this
21:55:15  <substack>I think generally speaking this is the answer to the "npm modules aren't discoverable" problem
21:55:20  <dominictarr>and it could all lead up to leveldb stuff
21:55:26  <substack>yep!
21:55:46  <substack>start out with simple in-memory ops, end at leveldb on disk map reducery
21:56:02  <thl0>Raynos: just see one render(main) at the bottom, but it renders other data in the middle as well like input
21:56:09  <dominictarr>intelligent curation of *-* ecosystems
21:56:11  <substack>a fantastic journey
21:56:17  <thl0>Raynos: or is that because it keeps changing?
21:56:39  * yorickquit (Remote host closed the connection)
21:56:51  <substack>dominictarr: it's like lord of the rings
21:56:55  <thl0>Raynos: really neat!, kindof more like lighttable
21:57:06  <substack>you go from place to place meeting all sorts of colorful characters and foes
21:57:11  <dominictarr>substack: actually, start with JSONStream, like your streams talk
21:57:20  <substack>but there is an over-arching narrative and purpose
21:57:44  <substack>narrative structure is too often missing from technical documentation
21:57:57  <dominictarr>you set it up so nicely, with a insurmountable problem, and then surmount it
21:58:38  <substack>yes exactly
21:58:52  <dominictarr>that sets the hook, and now they want to hear more
21:59:03  * defunctzombiechanged nick to defunctzombie_zz
21:59:03  <substack>hmm that's a good idea for my node philly talk too
21:59:12  <dominictarr>then, you show them how easy it is to write a stream
21:59:12  <substack>I'll start out by running procs in screen
21:59:32  <substack>and show how much of a pain that is to push new code, especially when you have multiple people on your team
21:59:44  <dominictarr>and not until THEN do you show them the first hard part - pause-stream
21:59:56  <substack>yep
22:00:15  <substack>fighting some golems
22:00:32  <substack>you don't get to that part until after you've met gandalf
22:01:05  <dominictarr>yup, and when you defeat them, and maybe another thing, and show them something REALLY cool, that they wouldn't even understand if you showed them it first.
22:02:16  <dominictarr>so, you begin with failing to open a 800 mb json file
22:02:50  <dominictarr>and then you end with building a clone of dynamo db, or a deployment system with realtime metrics
22:04:33  * defunctzombie_zzchanged nick to defunctzombie
22:04:44  <substack>it makes the case from the beginning about why you use a database in the first place
22:04:55  <substack>because JSON.parse() by itself doesn't scale
22:05:00  <dominictarr>yes
22:05:08  <substack>and it's important to start out with the assumptions if you're going to challenge some of them
22:05:40  <dominictarr>yes, but you never talk about the assumptions as such. avoid discussing theory until after
22:06:33  <substack>you can talk about them early so long as they are concrete
22:06:37  <substack>like an 800 mb json file
22:06:40  <dominictarr>yes
22:06:50  <dominictarr>everyone has that problem
22:06:53  <substack>specific -> general is always best
22:07:21  <substack>when theory comes first there's nothing to ground the ideas
22:07:40  <dominictarr>okay, so the character in the story is trying to open this massive file
22:07:48  <dominictarr>but his computer keeps crashing
22:07:55  * defunctzombiechanged nick to defunctzombie_zz
22:08:16  <dominictarr>then, a stranger wonders passed hears his problem
22:08:41  <dominictarr>and tells the kid about JSONStream
22:09:03  <substack>do this tech talk as a play
22:09:13  <substack>with multiple live action characters
22:09:28  <substack>that could be part of the screencast too
22:09:53  <dominictarr>okay -- so it turns out that the guy is an alien, who is stuck on earth, because his spaceship is broken
22:10:09  <Raynos>dominictarr: what I mean is I tried to make generic map reduce building blocks for mongo
22:10:11  <Raynos>and its a nightmare
22:10:12  <dominictarr>and, can you help him fix it
22:10:22  <Raynos>I think having real js and real module system for in process map reducing is really powerful
22:10:30  <Raynos>I just need to get to a point where I can experiment with it
22:11:20  <Raynos>thl0: its right at the bottom! ( http://raynos.graphics-server.jit.su/full/mario.js )
22:11:36  <dominictarr>substack: I like the play idea
22:11:37  <thl0>yeah, I saw that
22:11:37  <substack>dominictarr: the json file could be a star map
22:11:43  <dominictarr>yes
22:11:44  <substack>oh yes this is perfect
22:11:50  <thl0>but there are other things rendered in the middle - i.e. mario state
22:11:55  <substack>so the first thing he needs to do is to load the star map
22:12:01  <thl0>(object literals with numbers)
22:12:02  <substack>to figure out where he is
22:12:07  <Raynos>thl0: oh I see
22:12:23  <Raynos>thl0: put the mouse cursor over one of those white text blocks
22:12:30  <substack>but then once he's in space he needs to make quick course corrections
22:12:31  <Raynos>it says `inspect(dimensions) // =>`
22:12:40  <substack>which requires indexing
22:12:40  <Raynos>the => rendering engine just replaces the line with the value
22:12:40  <thl0>ah - got it
22:13:05  <thl0>pretty nifty
22:13:10  <dominictarr>substack: yes, and also to fight bitrot
22:13:32  <thl0>Raynos: I assume you have seen this: http://www.lighttable.com/ ?
22:13:32  <Raynos>thl0: this only really works for reactive style things
22:13:43  <Raynos>yeah ive seen lighttable I want to use it but it reeks of clojure
22:13:44  <thl0>Raynos: I see
22:13:48  <dominictarr>or some evil alien race that uses loc/programmer/day based metrics
22:14:17  <thl0>Raynos: :) thought the same thing, a buddy of mine used it for a while, but wished for his editor back rather soon
22:14:18  <substack>dominictarr: and then his spaceship adds new stars from its star detectors
22:14:27  <substack>so to make course corrections he'll also need to do inserts
22:14:34  <dominictarr>yeah, he has to send out probes
22:14:48  <thl0>Raynos: cool idea though, changes the way you'd think about code and how you organize it
22:14:49  <dominictarr>that must communicate with each other and pass back information
22:14:53  <substack>and sometimes the detectors will update star records with more precise positions
22:15:00  <substack>or to correct errors or sometimes stars move
22:15:13  <dominictarr>substack: also, stuff like fuel sources
22:15:32  <substack>gas giants could be the fuel sources
22:15:38  <dominictarr>and because of the great distances involved in interstellar travel, you have to be aware of concurrent updates
22:15:40  <substack>to power hydrogen fusion
22:15:53  <substack>yep!
22:16:02  <dominictarr>also
22:16:08  <substack>also when you're going at full speed you lose radio communication
22:16:10  <substack>network partitions
22:16:20  <dominictarr>absolutly!
22:16:25  <substack>because of radio noise from the plasma fusion core
22:16:37  <dominictarr>and your travelling at lightspeed!
22:17:17  <dominictarr>substack: so this alien has to get back to his home planet, to deliver a report he's written on earth
22:17:58  <substack>exactly the same as ford prefect
22:18:06  <substack>harmless -> mostly harmless
22:18:10  <dominictarr>yes, but different report
22:18:25  <substack>in fact you could make it about the hitchhiker's guide
22:18:31  <dominictarr>the report says: "Humans are idiots, and no good at writing software"
22:18:57  <substack>the hhg has concurrent document updates and network partitions
22:19:09  * tmcw_quit (Remote host closed the connection)
22:19:19  <dominictarr>continues "although, the previous U.N.I.X mission was successful, not enough people understand it's principles"
22:19:42  <substack>hahaha that's like U.N.I.T. from dr who
22:19:45  <Raynos>thl0: your scriptie talkie is confusing
22:20:01  <Raynos>mainly because i dont care about the return value of every line
22:20:21  * tmcwjoined
22:20:31  <Raynos>thl0: although having https://gist.github.com/Raynos/b8bf27d5d05811858bfc#file-continuable-js-L15 the type comments auto generated would be bad ass
22:21:42  <dominictarr>"the humans have rested on their laurells and not improved on the start we've given them"
22:21:42  <rowbit>Hourly usage stats: [developer: 4, free: 34]
22:22:05  <thl0>Raynos: yeah it's not for all use cases, mainly to debug your js and understand what it does every step of the way
22:22:15  <thl0>Raynos: or to teach people js via example
22:22:17  <dominictarr>"recommend earth for preemptive destruction, unless they join the bitrot hoard"
22:22:29  <Raynos>then it needs better syntax
22:22:31  <substack>garbage collection
22:22:37  <thl0>I thought it would be great to show examples in a blog
22:22:43  <thl0>Raynos: like how ?
22:22:44  <dominictarr>rm -rf /earth
22:22:53  <thl0>so far it uses difflet to show changes
22:22:54  <substack>garbage collect earth to save space
22:23:07  <thl0>pretty clear syntax in my op
22:23:11  <dominictarr>faster reads in that quadrant of the dataverse
22:23:14  <Raynos>not sure
22:23:17  <Raynos>The thing is
22:23:19  <Raynos>I want less information
22:23:27  <Raynos>I dont know how to render "enough" information
22:23:28  <dominictarr>substack: anyway, you find that out in the middle
22:23:39  <thl0>Raynos: got it - that is a tough one
22:24:01  <thl0>cause either you show only last OR all intermediate results
22:24:16  <dominictarr>when the alien and the kid are both on the way to the home planet
22:24:20  * wolfeidauquit (Remote host closed the connection)
22:24:24  <thl0>that's why I didn't like slowmoJS because it repeated to show things that didn't even change
22:24:41  <thl0>at least scriptie-talkie only shows changes in the context
22:25:14  <dominictarr>maybe, the kid learn of this without the alien realizing
22:25:16  <Raynos>thl0: like in my case. I'm rendering values over time and only the ones I care about
22:25:22  <Raynos>ideally it's toggleable
22:25:28  <Raynos> ideally it has a timeline
22:25:32  <Raynos>I havn't gotten that far yet
22:25:43  <thl0>Raynos: right, maybe I could add an option to ignore a snippet result/change
22:25:43  * ins0mniaquit (Ping timeout: 260 seconds)
22:25:50  <dominictarr>anyway, so they have a bunch more adventures after that -
22:26:10  <thl0>Raynos: someting like // ignore at the end
22:26:39  <thl0>at least for the embed version
22:26:49  <dominictarr>and just before they reach the alien planet - the alien realizes that the kid (and reader) has learnt all the principles - so maybe there is hope for earth - and he decides not to hand in the report
22:26:56  <thl0>for the main version it's totally fine to see a bit more than you want
22:27:43  <dominictarr>substack: maybe there can be spacestations they stop at and fix along the way too
22:29:34  * shuaibjoined
22:39:13  <dominictarr>substack: idea: ssh a node repl into any browser
22:40:03  * jcrugzzquit (Quit: leaving)
22:40:07  <dominictarr>want to go document.body.append(document.createTextNode('HI THERE'))
22:40:18  <dominictarr>in the terminal and see it the a browser
22:41:09  * jcrugzzjoined
22:41:33  <substack>shoe('/sock').pipe(split()).pipe(through(function (line) { Function(line)() })
22:43:59  * wolfeidaujoined
22:44:26  * farnsworthquit (Remote host closed the connection)
22:44:26  * cubertquit (Remote host closed the connection)
22:48:14  * shuaibquit (Ping timeout: 252 seconds)
22:51:27  <dominictarr>hmm, that is gonna make a good demo
22:51:53  <dominictarr>what about with auto complete?
22:51:57  <chrisdickinson>thl0: belatedly: very neat re: scriptie-talkie embeds
22:52:15  <substack>dominictarr: 8000% more complicated with auto-complete
22:52:39  <substack>just do the simple easy thing
22:53:01  <dominictarr>yes
22:53:11  <thl0>chrisdickinson: thnx - still working on adding some features
22:53:17  <chrisdickinson>anyone have good primitives for an api that is *usually* synchronous but occasionally async?
22:53:31  <chrisdickinson>with an eye towards optimizing towards the sync case?
22:53:46  <dominictarr>chrisdickinson: you mean, callsback before it returns?
22:53:47  <chrisdickinson>(nb, just making everything async is what i'm already doing and it's horrible for performance.)
22:54:09  <chrisdickinson>dominictarr: in this case it's a through stream that implements inflate
22:54:27  <chrisdickinson>the problem is that currently i have an async api for asking for a certain number of bits
22:54:33  <dominictarr>https://github.com/dominictarr/pull-stream/blob/master/sinks.js#L1-L22
22:54:35  <chrisdickinson>(like pull-streams)
22:54:39  <dominictarr>did that yesterday
22:54:49  <chrisdickinson>well
22:55:05  <chrisdickinson>so here's the problem; you can't recurse because of a lack of tco
22:55:11  <dominictarr>it checks if it's calledback sync, and iterates instead of recursing
22:55:19  <chrisdickinson>if you just recurse it'll reliably throw recursion errors
22:55:22  <dominictarr>tco?
22:55:28  <chrisdickinson>tail call optimization
22:55:48  * thl0quit (Remote host closed the connection)
22:55:56  <dominictarr>chrisdickinson: that is what the link does - iterates if can,
22:55:56  * tilgovijoined
22:56:04  <dominictarr>else recurses if it's async
22:56:42  <chrisdickinson>yes, but i'm asking for a bit at a time, nearly
22:56:49  <chrisdickinson>so it'll call done and go to the next step
22:57:00  <chrisdickinson>which may or may not ask for 1+ bits
22:57:36  <chrisdickinson>https://gist.github.com/chrisdickinson/a3a9f947ce586a272a9b
22:57:55  <chrisdickinson>things are unrolled a bit to be nicer to V8 for optimization
22:58:13  <chrisdickinson>basically you can either be in "read a byte" mode or "read N bits" mode
22:58:22  <chrisdickinson>take === "read bits", read === "read bytes"
22:58:24  <dominictarr>substack: what if you replicated the current variables back to the repl?
22:58:27  <chrisdickinson>https://gist.github.com/chrisdickinson/a3a9f947ce586a272a9b#file-inflate-js-L230-L233
22:58:40  <substack>dominictarr: with scuttlebutt?
22:58:44  <dominictarr>yeah
22:58:50  <substack>that would be really neat
22:58:52  <dominictarr>would make it way more snappy anyway
22:59:45  <dominictarr>and you only need to support thing.property.whatever
23:00:34  <chrisdickinson>prevailing wisdom seems to be "if anything's async make it all async"
23:00:35  <dominictarr>maybe with some trick to remember variables you have created
23:01:20  <chrisdickinson>but there's kind of a corner case where things will *usually* be sync, and delegating to process.nextTick / setImmediate adds far too much overhead
23:02:18  <chrisdickinson>was having a similar problem with plate when i was trying to make it not dog-slow -- and i used something that i called a "promise" but isn't really a promise
23:02:23  <chrisdickinson>https://github.com/chrisdickinson/plate/blob/master/lib/tags/with.js#L24-L47
23:02:46  <chrisdickinson>basically, try to do everything synchronously until you get a value that indicates that you have to wait. once that gets resolved, retry.
23:03:16  <chrisdickinson>just curious if this corner case has come up for anyone else, and if so how have you tackled it?
23:04:33  * shuaibjoined
23:04:46  <dominictarr>chrisdickinson: yeah, don't use nextTick
23:05:26  <chrisdickinson>yes, but then you get into the problem of getting recursion errors
23:05:59  <dominictarr>you need to make your control flow work with this : function fakeAsyncMaybe(data, cb) { if(maybe) cb() /*immediate*/; else process.nextTick(cb) })
23:06:04  * tmcwquit (Remote host closed the connection)
23:06:26  <dominictarr>so, I had that… and SOMETIMES i uses nextTick...
23:06:45  <dominictarr>like if(!(i++ % 1000)) nextTick(cb) else cb()
23:08:19  <dominictarr>but then I switched to a different approach
23:08:39  <dominictarr>like - have take check whether it has been called before an outer take has returned
23:08:57  <chrisdickinson>yeah, that's what i'm doing now, roughly
23:08:58  <dominictarr>and if so, add your callback to the stack
23:09:10  <dominictarr>and let the outer stack call it
23:09:26  <chrisdickinson>i think maybe it might be best to go for "return a sentinel value" and have resumable state in the stream
23:09:35  <chrisdickinson>not quite the promise solution from before
23:09:58  <dominictarr>I did that yesterday (in one place for a long pull stream) and it went 20% faster
23:10:37  <chrisdickinson>interesting, so you end up with a stack of callbacks that get iterated over?
23:10:51  <substack>use 1 mdm stream for a scuttlebutt model and 1 mdm stream for a through(function (line) { model.set('_result', Function(model.keys(), line).apply(null, model.keys().map(function (key) { return model.get(key) })))) })
23:10:58  <substack>and pipe the mdm stream through shoe
23:11:13  <substack>then on the other end hook into updates to _result and console.dir() those values
23:11:15  <dominictarr>in my case it was just a loop, not full recursion, so I didn't need a stack
23:11:27  <chrisdickinson>substack: sidenote: does shoe/sock support binary data (i.e., sending arraybuffers)?
23:11:35  <dominictarr>substack: yes, that would work
23:11:36  <substack>chrisdickinson: use base64
23:11:44  <substack>it's only 33% more data
23:12:08  <chrisdickinson>substack: would you be greatly opposed to a PR that added support for arraybuffers?
23:12:10  <jesusabdullah>but think of the dial-up users!
23:12:19  <dominictarr>chrisdickinson: … some of the fallbacks might break?
23:12:33  <chrisdickinson>dominictarr: true
23:12:39  <chrisdickinson>the flash one, primarily
23:12:46  <chrisdickinson>(if that one exists in sock)
23:12:46  <dominictarr>maybe you could degrade intelligently, though
23:12:51  <st_luke>I send base encoded images over websockets all the time, works quite well
23:13:29  <chrisdickinson>i've been using base64
23:13:46  <dominictarr>chrisdickinson: mostly sock js is something that I have been able to avoid learning how it actually works.
23:13:51  <chrisdickinson>it's adding a bit of overhead. outside of this "maybe possibly async" thing it's the second bottleneck
23:14:25  <dominictarr>chrisdickinson: I've been thinking about a js implemention of snappy
23:14:41  <chrisdickinson>interesting!
23:14:48  <dominictarr>it's byte (not bit) compression optimized for speed
23:15:15  <substack>chrisdickinson: I kind of don't care about arraybuffers
23:15:25  <substack>base64 and everything works everywhere
23:15:30  <substack>worse is better here
23:15:38  <dominictarr>& there is lots of reppetittion in json/mux-demux/scuttlebutt so there would be pretty good gains if it was fast enough
23:15:40  <jesusabdullah>turkey salad :D
23:15:53  <chrisdickinson>i'm pretty tied to inflate (it's baked into the git packfile spec), but it'd definitely be nice to have more options
23:16:10  <dominictarr>yes - inflate is part of git
23:16:14  <jesusabdullah>infl8 huh?
23:16:56  <chrisdickinson>substack: yeah, for the moment solving the recursion / nextTick thing in inflate will buy me back a bunch of time, so it remains to be seen if the base64->buffer bottleneck will absolutely *need* solving
23:17:20  <dominictarr>chrisdickinson: https://gist.github.com/chrisdickinson/a3a9f947ce586a272a9b#file-inflate-js-L96-L115
23:17:28  <dominictarr>is that rolled out for perf?
23:17:40  <chrisdickinson>yep
23:17:50  <dominictarr>does that make much difference?
23:17:55  <chrisdickinson>it's a little dumb but it bought me about 200ms on a 1.6mb file
23:18:02  <chrisdickinson>200-300-ish
23:18:21  <dominictarr>how much did it do before hand?
23:18:36  <dominictarr>I mean, how long did it take first
23:18:46  <dominictarr>s/first/before/
23:18:46  <chrisdickinson>so, this is before the biggest speed gain, which was hilariously located elsewhere
23:19:19  <chrisdickinson>iirc it was like 1.4-6 before, 1.2-3 after
23:19:22  <chrisdickinson>but
23:19:30  <dominictarr>1.4 seconds?
23:19:35  <chrisdickinson>yeah
23:19:53  <chrisdickinson>alarming, but, there was another fix that bought a ton of time
23:20:08  <dominictarr>what was that?
23:20:13  <chrisdickinson>https://gist.github.com/chrisdickinson/a3a9f947ce586a272a9b#file-inflate-js-L403-L419
23:20:14  <chrisdickinson>so
23:20:20  <chrisdickinson>you have to keep a window around
23:20:35  <chrisdickinson>i am dumb, and i adding to the array in the loop and then slicing it to 32k at the end
23:20:59  <chrisdickinson>which really meant that it was copying and GC'ing 32k every time there was output
23:21:03  <dominictarr>oh, forcing types to int
23:21:05  * thl0joined
23:21:37  <dominictarr>… and is 32k a magic number, memory wise?
23:21:50  <chrisdickinson>it's the size of the largest window necessary to inflate a deflated stream
23:22:12  <chrisdickinson>you can be smarter about it, theoretically -- i do believe the window size is part of the header
23:22:30  <chrisdickinson>but there's also raw deflated streams which don't have the header, so i opted for simplicity
23:23:30  <dominictarr>what if you made read async, but take sync?
23:23:43  <chrisdickinson>well, take can take >1 bit
23:23:46  <chrisdickinson>>=1 bit
23:23:51  <dominictarr>so you read to a hwm and then take
23:23:51  <chrisdickinson>somewhere between 1 and 20, iirc
23:24:16  <chrisdickinson>and you might be on a byte boundary, and that byte boundary might also be a buffer boundary, and then you end up having to wait
23:24:24  <dominictarr>right - so if you have less than 20 bits in the stream, you should probably do a read
23:24:35  <chrisdickinson>right
23:24:56  <chrisdickinson>but the other issue is that when you're doing the codes, you have no idea how many bits you'll actually have to take in this loop
23:25:10  <dominictarr>what if you just started the last take over if necessary
23:25:14  <chrisdickinson>you have to take until you recurse down the huffman tree to a end symbol
23:25:23  <chrisdickinson>*an end
23:25:36  <dominictarr>but how long is each symbol, usually?
23:25:44  <dominictarr>it's gonna be pretty short, right?
23:26:02  <chrisdickinson>well, the symbol is generated by using the bits to recurse down the huffman tree (as i understand it)
23:26:10  <dominictarr>less than 10 bytes etc
23:27:28  <chrisdickinson>the symbol just tells you what to do next (symbol < 256, output raw symbol, symbol > 256, use lencodes/distcodes to copy back out of the output window, symbol === 256, end block)
23:28:24  <dominictarr>right, so what is the typical length for a block?
23:28:48  <dominictarr>vs a read?
23:29:10  <dominictarr>a read is probably a block at a time
23:29:19  <dominictarr>4k or so
23:31:07  <dominictarr>even if the input is completely random, you are gonna get many inflate blocks for one os block… surely?
23:32:21  * mikolalysenkojoined
23:42:32  * stlsaintjoined
23:44:33  * fallsemoquit (Quit: Leaving.)
23:45:37  * spionquit (Ping timeout: 246 seconds)
23:45:53  * fallsemojoined
23:46:00  <chrisdickinson>dominictarr: i haven't measured the typical block length, but they're definitely not byte aligned
23:46:36  <chrisdickinson>there are 3 block types - stored, fixed, and dynamic -- stored is the simplest and is byte aligned after the header (it's just 0-65535 bytes)
23:47:18  <chrisdickinson>fixed and dynamic both use huffman coding to generate output -- dynamic actually transmits the serialized huffman codes in-stream though
23:47:19  <dominictarr>so, what I'm wondering, is could you just redo blocks that turned out to overlap?
23:47:48  <chrisdickinson>i don't think there's any one-to-one between a buffer coming in representing a single block
23:47:53  <dominictarr>those being the rare cases that are async?
23:47:55  <chrisdickinson>it could represent <1, ==1, or >1 blocks
23:48:07  <chrisdickinson>hmm
23:48:18  <chrisdickinson>possibly
23:48:19  <dominictarr>so.. blocks could be larger than blocks?
23:48:37  <chrisdickinson>blocks could be larger or smaller than the incoming buffer data yes
23:48:38  <dominictarr>buffers could be larger than blocks, I mean
23:48:50  <chrisdickinson>yep
23:49:02  <chrisdickinson>there's no one-to-one, especially since the headers etc aren't byte aligned
23:49:27  <dominictarr>hmm, so what is the case that generates the longest blocks? random data - or data with repitions?
23:49:30  <chrisdickinson>this is tangentially the problem with node's zlib -- it'll give you data before it gets to the (byte aligned) 4 byte adler at the end
23:50:08  <chrisdickinson>random data would be stored byte-aligned as a literal "stored" block
23:50:42  <dominictarr>right - so that isn't a problem
23:51:29  <dominictarr>hmm - just checking - this is the decompressor we are talking about, right?
23:51:35  <chrisdickinson>yes
23:52:09  <dominictarr>how big is the huffman tree?
23:53:34  <dominictarr>hmm, so basically the approach I am suggesting is this https://github.com/dominictarr/split/blob/master/index.js
23:53:48  <dominictarr>process it sync
23:53:52  <chrisdickinson>i think it varies
23:53:58  <dominictarr>but keep the part that didn't make sense, and try it again
23:54:45  <chrisdickinson>ah
23:54:50  <chrisdickinson>and then retry it when we get more data
23:54:54  <chrisdickinson>interesting
23:59:13  <dominictarr>you'll probably won't waste much time retrying, if you receive reasonable sized buffers