00:00:01  * ircretaryquit (Remote host closed the connection)
00:00:10  * ircretaryjoined
00:25:03  * jcrugzzquit (Ping timeout: 240 seconds)
00:33:22  * toddselfjoined
00:34:43  * toddselfquit (Remote host closed the connection)
00:49:35  * ferossquit (Quit: feross)
00:50:54  * thlorenzjoined
01:05:37  <mbalho>csv test data: https://github.com/maxogden/csv-spectrum
01:05:57  <mbalho>cc juliangruber rvagg (any any other csv parser authors in here that i'm aware of)
01:22:38  <ralphtheninja>dominictarr: https://www.youtube.com/watch?v=2_XLdBd1tMY
01:25:57  * venportmanquit (Read error: Connection reset by peer)
01:33:36  <mbalho>brianloveswords: jlord requests a balrog push
01:34:57  * ralphtheninjaquit (Read error: Operation timed out)
01:42:43  * ferossjoined
02:03:01  * ferossquit (Quit: feross)
02:18:13  <jesusabdullah>mbalho: cool to see that github seems to pass the tough ones
02:19:09  <jesusabdullah>pass all of them, really
02:19:11  <jesusabdullah>g'deal
02:22:39  * mikolalysenkojoined
02:26:17  <mbalho>they strip newlines and also don't render the latin1 character correctly but those dont really matter
02:30:02  * defunctzombiechanged nick to defunctzombie_zz
02:31:18  * mikolalysenkoquit (Ping timeout: 264 seconds)
02:37:51  <jlord>haha thanks mbalho
02:39:34  <jlord>brianloveswords: index.js needs to be reworked a bunch to fit the pagination in but I don't want to redo anything you've done and I think you're already reworking it. can you push and I can start from there?
02:39:48  * mikolalysenkojoined
02:48:28  * jxsonjoined
02:58:47  * mikolalysenkoquit (Ping timeout: 260 seconds)
03:12:09  * defunctzombie_zzchanged nick to defunctzombie
03:15:39  * thlorenzquit (Remote host closed the connection)
03:26:28  * jxsonquit (Remote host closed the connection)
03:29:14  * nlacassequit (Ping timeout: 245 seconds)
03:29:14  * philipnquit (Ping timeout: 245 seconds)
03:29:15  * grncdr_quit (Ping timeout: 245 seconds)
03:29:16  * philipn_joined
03:29:19  * nlacasse_joined
03:29:21  * grncdrjoined
03:34:35  <jesusabdullah>mbalho: how do you feel about tsv?
03:35:12  * jxsonjoined
03:44:00  * ferossjoined
04:04:00  * jxsonquit (Remote host closed the connection)
04:17:18  * ferossquit (Quit: feross)
04:20:48  * shamaquit (Remote host closed the connection)
04:21:22  * jxsonjoined
04:38:10  * gwenbellquit (Remote host closed the connection)
04:43:53  <mbalho>jesusabdullah: i approve of all tabular data formats
04:44:25  <jesusabdullah>nice
05:00:13  * defunctzombiechanged nick to defunctzombie_zz
05:07:52  * timoxleyjoined
05:31:38  * michaelrhodesjoined
05:32:56  * michaelrhodesquit (Quit: Leaving)
05:33:24  * jxsonquit (Remote host closed the connection)
05:38:08  * jxsonjoined
05:38:13  * jxsonquit (Remote host closed the connection)
05:49:15  * jxsonjoined
05:57:08  * jxsonquit (Remote host closed the connection)
05:58:43  * jxsonjoined
06:15:22  * jergasonjoined
06:31:29  * thlorenzjoined
06:32:57  * kriskowaljoined
06:34:32  * kriskowalquit (Client Quit)
06:37:39  * jergasonquit (Quit: jergason)
06:39:43  * thlorenzquit (Ping timeout: 245 seconds)
06:41:16  * jcrugzzjoined
06:50:04  * jxsonquit (Remote host closed the connection)
06:52:14  * jxsonjoined
07:05:13  * jxsonquit (Remote host closed the connection)
07:07:10  * jxsonjoined
07:40:41  * alessioalexjoined
08:15:43  * stagasjoined
08:16:17  * jxsonquit (Remote host closed the connection)
08:30:35  * jcrugzzquit (Ping timeout: 260 seconds)
09:04:20  * ralphtheninjajoined
09:08:32  * ralphtheninjaquit (Client Quit)
09:08:40  * ralphtheninjajoined
09:17:27  * thlorenz_joined
09:21:42  * thlorenz_quit (Ping timeout: 252 seconds)
09:23:34  * jxsonjoined
09:27:04  * alessioalexquit (Quit: Page closed)
09:28:18  * jxsonquit (Ping timeout: 256 seconds)
09:34:51  <dominictarr>juliangruber: hey, remember those level vs mongo vs redis benchmarks you did? where are they?
09:35:23  <dominictarr>nm, found them
09:36:59  * jcrugzzjoined
09:41:08  * jcrugzzquit (Ping timeout: 240 seconds)
09:56:38  * dominictarrquit (Quit: dominictarr)
10:02:02  * Kesslerjoined
10:02:16  * dominictarrjoined
10:24:19  * jxsonjoined
10:30:08  * Kessler_joined
10:30:29  * Kesslerquit (Ping timeout: 248 seconds)
10:32:10  * jxsonquit (Read error: Operation timed out)
11:18:06  * thlorenzjoined
11:22:46  * thlorenzquit (Ping timeout: 256 seconds)
11:28:59  * Birdbonesquit (Ping timeout: 248 seconds)
11:43:13  * stagasquit (Read error: Connection reset by peer)
12:09:23  * thlorenzjoined
12:37:07  * st_lukejoined
12:40:22  * Kesslerjoined
12:42:29  * Kessler_quit (Ping timeout: 256 seconds)
12:43:51  * Kessler_joined
12:46:54  * Kesslerquit (Ping timeout: 264 seconds)
12:53:22  * Kesslerjoined
12:55:58  * Kessler_quit (Ping timeout: 245 seconds)
12:59:33  * st_lukequit (Remote host closed the connection)
13:07:55  * Guest8803joined
13:08:16  * Guest8803quit (Remote host closed the connection)
13:09:27  * fuzjoined
13:14:29  * st_lukejoined
13:26:48  * Birdbonesjoined
13:46:15  * yorickjoined
13:57:15  * fuzquit (Ping timeout: 252 seconds)
13:58:14  * st_lukequit (Remote host closed the connection)
14:15:10  * jergasonjoined
14:24:37  * fuzjoined
14:25:01  * fuzchanged nick to Guest10038
14:26:04  * Guest10038quit (Client Quit)
14:26:21  * fuz__joined
14:26:49  * fuz__changed nick to fuz
14:34:05  * Kesslerquit (Quit: https://github.com/kessler)
14:34:20  * Kesslerjoined
14:45:25  * fuzquit (Ping timeout: 248 seconds)
14:59:10  * jergasonquit (Quit: jergason)
15:19:35  * mikolalysenkojoined
15:24:10  * mikolalysenkoquit (Ping timeout: 240 seconds)
15:26:37  * defunctzombie_zzchanged nick to defunctzombie
15:34:23  * mikolalysenkojoined
15:58:58  * dominictarrquit (Quit: dominictarr)
16:01:53  * anoemijoined
16:08:03  * AvianFlujoined
16:13:07  * shamajoined
16:22:32  * mikolalysenkoquit (Ping timeout: 256 seconds)
16:32:00  <mbalho>dat has a logo now! http://dat-data.com/ now i just need to finish coding the damn thing :D
16:32:29  <mbalho>(theres an easter egg in the logo, probably obvious)
16:34:52  * yorickquit (Remote host closed the connection)
16:44:52  * dominictarrjoined
16:48:17  * dominictarrquit (Client Quit)
16:51:04  * dguttmanjoined
16:54:04  * dominictarrjoined
16:55:34  * kenperkinsquit (Quit: Computer has gone to sleep.)
16:56:25  * dguttmanquit (Quit: dguttman)
17:10:25  * chiltsquit (Ping timeout: 245 seconds)
17:11:55  * AvianFluquit (Remote host closed the connection)
17:13:54  * AvianFlujoined
17:14:11  * thlorenzquit (Remote host closed the connection)
17:14:13  * Kesslerquit (Ping timeout: 248 seconds)
17:14:21  * timoxleyquit (Remote host closed the connection)
17:16:18  * Kesslerjoined
17:16:22  * stagasjoined
17:17:31  * anoemiquit (Quit: anoemi)
17:18:07  * chiltsjoined
17:28:04  * mikolalysenkojoined
17:28:21  * chiltsquit (Ping timeout: 245 seconds)
17:31:40  * Kesslerquit (Ping timeout: 256 seconds)
17:34:38  * Kesslerjoined
17:34:40  <shama>mbalho: dat is a beautifully logo
17:37:33  <mbalho>thx jlord made it
17:43:04  <shama>I totally dont see the easter egg
17:44:25  <mbalho>shama: d a t
17:47:26  <shama>im staring at it and now it looks like a voxel cube and I cant unsee the cube now heh
17:47:53  <mbalho>shama: the three triangles on the right are the letters D, A, T :D
17:47:57  <mbalho>shama: also in the logo twice
17:48:21  <mbalho>shama: i cant see the cube though, will keep lookin
17:48:28  <shama>:)
17:48:29  <mbalho>OH
17:48:32  <mbalho>whoa
17:48:32  <shama>haha
17:48:42  <shama>now you wont be able to unsee
17:49:01  <mbalho>lol
17:51:40  * chiltsjoined
17:52:17  * kriskowaljoined
17:57:07  * mikolalysenkoquit (Ping timeout: 260 seconds)
17:57:25  * dominictarrquit (Ping timeout: 248 seconds)
17:58:07  * dominictarrjoined
17:59:25  * jergasonjoined
17:59:55  * chiltsquit (Ping timeout: 240 seconds)
18:01:15  * timoxleyjoined
18:04:18  * jergasonquit (Ping timeout: 264 seconds)
18:05:26  * timoxleyquit (Ping timeout: 245 seconds)
18:08:37  * defunctzombiechanged nick to defunctzombie_zz
18:09:51  * anoemijoined
18:23:39  * defunctzombie_zzchanged nick to defunctzombie
18:25:31  <mbalho>trevnorris: is there any way, between one node process piping to another, to force stdin on the receiver to emit the exact buffer that was written to stdout in the sender? default behavior is to buffer subsequent stdout writes into a larger buffer
18:26:09  <mbalho>trevnorris: my use case is that I wanna do something like csv-parser some_huge_csv.csv | store-in-database
18:26:37  * st_lukejoined
18:26:57  <mbalho>trevnorris: where csv-parser and store-in-database are both node programs, and csv-parser splits the csv into rows, writes each row (each row is a Buffer) to stdout, then store-in-database puts each buffer in the database
18:27:40  <mbalho>trevnorris: i can combine csv-parser and store-in-database into one bigger program but i'd like to keep them separate if possible. i guess what i'm asking for is if theres a performant way to share/copy buffers between two processes
18:30:11  * tmcwjoined
18:34:27  * Kesslerquit (Ping timeout: 260 seconds)
18:40:53  * chiltsjoined
18:45:26  <dominictarr>mbalho: what is the dat logo?
18:45:31  * dguttmanjoined
18:46:08  <dominictarr>mbalho: the operating system decideds where the edges of buffers would be
18:46:21  <dominictarr>if you don't like that you need to use a framing protocol
18:47:58  <mbalho>dominictarr: yea i figured, but i didnt know if there were any special tricks
18:49:17  <mbalho>dominictarr: i think instead of trying to frame the data myself i'm just going to move more complicated parsers like csv and json into dat core, as well as a straightforward newline parser
18:49:48  <mbalho>dominictarr: so that all the parsing and storage happens in the same process. a complexity for speeds sake at the expense of CLI modularity
18:49:58  <dominictarr>sounds reasonable
18:50:05  <dominictarr>you could also use redis protocol!
18:50:11  <mbalho>dominictarr: also if you can get your data into a newline separated form then you can pipe it in from another process
18:50:20  <dominictarr>that has a 1:1 with csv
18:50:23  <dominictarr>and does binary
18:50:35  <dominictarr>and is standard and framed
18:50:49  <dominictarr>there is even a streaming module for it
18:50:53  <mbalho>dominictarr: i'll look into it, thanks
18:51:04  <dominictarr>https://npm.im/redis-protocol-stream
18:51:09  <mbalho>dominictarr: i basically just need [Buffer, Buffer, Buffer, Buffer] between processes
18:51:27  <dominictarr>that is what redis protocol is
18:51:39  <mbalho>i could probably just use \xff or somethign as my delimiter instead of \n and be fine for most things
18:52:18  <dominictarr>well, with a framed format you don't have to iterate over the data
18:52:40  <dominictarr>you just read the length, then copy everything until that
18:53:12  <dominictarr>if you have a char, you have to check whether each char is the delimiter
18:53:27  <mbalho>dominictarr: good point. i'll spike it out, do some benchmarking
18:53:32  <dominictarr>also, escaping...
18:53:32  <dominictarr>so length delimited is more scalable
18:53:40  <dominictarr>cool
18:54:07  <dominictarr>mbalho: what is the dat logo?
18:54:19  <mbalho>dominictarr: you want to see it? http://dat-data.com
18:54:44  <mbalho>dominictarr: its a hexagon made up of abstract geometric represntations of the letters D A T
18:55:13  <mbalho>dominictarr: hmm sucks that its require('redis/lib/parser/javascript').Parser and not require('redis-parser-js') or something
18:55:28  * thlorenzjoined
18:56:59  <dominictarr>yes, maybe they'd take a pr?
18:57:23  <dominictarr>I think it was written before substack had convinced everyone about modules
18:57:36  <mbalho>lol
18:58:43  <dominictarr>you logo looks very aquireable. here is a suitcase full of money!
18:59:14  <mbalho>:)
19:01:49  * timoxleyjoined
19:03:42  * mikolalysenkojoined
19:06:18  * timoxleyquit (Ping timeout: 256 seconds)
19:07:56  * mikolalysenkoquit (Ping timeout: 245 seconds)
19:10:54  * st_lukequit (Remote host closed the connection)
19:12:20  * st_lukejoined
19:16:48  * st_lukequit (Ping timeout: 252 seconds)
19:21:15  * anoemiquit (Quit: anoemi)
19:22:43  * dominictarrquit (Quit: dominictarr)
19:23:12  <tmcw>is there a streaming json writer that's commonly used?
19:24:04  <mbalho>tmcw: JSONStream's stringifier i think
19:24:47  <mbalho>tmcw: or if you want ldjson theres ldjson-stream
19:25:00  <tmcw>awesome, y, looks like jsonstream does non-ld json
19:25:13  <tmcw>still kind of iffy on ld json - I get the simplicity of parsing it in some ways, but...
19:26:12  <mbalho>tmcw: basically if you control both sides its the simplest thing, and i think its nice to offer as an option
19:28:09  * anoemijoined
19:33:59  <jesusabdullah>non-ld?
19:34:03  <jesusabdullah>I see ld I think lds
19:34:22  <jesusabdullah>cause utah
19:43:36  * jcrugzzjoined
19:50:17  * tmcwquit (Remote host closed the connection)
19:57:46  * defunctzombiechanged nick to defunctzombie_zz
20:01:01  * defunctzombie_zzchanged nick to defunctzombie
20:02:40  * timoxleyjoined
20:03:35  * anoemiquit (Quit: anoemi)
20:07:06  * timoxleyquit (Ping timeout: 245 seconds)
20:08:00  * st_lukejoined
20:14:20  * dominictarrjoined
20:18:30  * AvianFluquit (Remote host closed the connection)
20:21:32  <defunctzombie>memoization lib?
20:22:29  * defunctzombiechanged nick to defunctzombie_zz
20:23:00  <dominictarr>rvagg: hey, when do you get to lisbon?
20:24:55  <trevnorris>mbalho: ping
20:29:54  * owen1quit (Quit: WeeChat 0.4.1)
20:30:39  * dominictarrquit (Ping timeout: 260 seconds)
20:34:33  * dominictarrjoined
20:43:49  * defunctzombie_zzchanged nick to defunctzombie
20:45:42  * orenchanged nick to owen1
20:49:04  * AvianFlujoined
20:52:28  <mbalho>trevnorris: yo
20:52:43  <trevnorris>mbalho: so what's going on?
20:52:55  <mbalho>trevnorris: after talking with dominictarr i am realizing that i probably need to implement my own framing in order to move buffers between two node processes
20:53:43  <mbalho>trevnorris: i wasnt sure if there was some secret way to exchange a set of buffers between two node processes without combining them (which is what stdout/stdin does)
20:54:07  <dominictarr>like a "direct memory swap"
20:54:37  <trevnorris>mbalho: I do believe that even IPC still requires a memcpy, but if all you're trying to do is write out buffers individually and don't care about the memcpy then yeah. it's doable
20:54:57  <dominictarr>(I don't think unix does that… would love to be corrected)
20:55:20  * st_lukequit (Remote host closed the connection)
20:56:45  <trevnorris>does what?
20:57:42  * AvianFluquit (Ping timeout: 264 seconds)
20:58:08  * kriskowalquit (Quit: kriskowal)
21:00:41  <trevnorris>mbalho: sorry, in between several things and have a hard time multi-tasking. :)
21:00:41  <trevnorris>why can't this operation be done on a separate thread?
21:01:04  * jergasonjoined
21:01:14  <mbalho>trevnorris: its for writing unix style command line utilities that do fast data processing
21:02:20  <trevnorris>mbalho: ok, so i'll assume they'll be doing data processing in js. hence the need for running separate processes.
21:03:18  * timoxleyjoined
21:04:10  <trevnorris>mbalho: so is it like, take in this file, split it by X and send it to processes A, B and C that are user defined scripts for actual processing?
21:07:53  <trevnorris>mbalho: this is master branch only, but the fastest way will be to overwrite the handles StreamWrapCallbacks::DoRead() in a native module. intercept the incoming data, do the memory split in memory.
21:07:53  <trevnorris>mbalho: then you'll want to intercept the StreamWrapCallbacks::DoWrite() method for the out going handle and send the data directly through there.
21:08:08  * timoxleyquit (Ping timeout: 256 seconds)
21:08:19  * defunctzombiechanged nick to defunctzombie_zz
21:08:37  <ralphtheninja>dominictarr: autonomous agents (mike hearn) http://www.youtube.com/watch?v=Pu4PAMFPo5Y&feature=youtu.be
21:08:40  * AvianFlujoined
21:09:10  <trevnorris>mbalho: you can template-ize this so the logic of where data comes/goes is all done in JS, but the data itself is only handles in C++.
21:09:42  <trevnorris>mbalho: but if you're looking for something a little less intense, let me know and I'll write you up an example script of what I _think_ you're trying to do. :)
21:13:44  * anoemijoined
21:14:35  * kriskowaljoined
21:19:22  <mbalho>trevnorris: nice, thats a good start for now, cheers
21:19:31  <mbalho>trevnorris: i mostly wanted to know if its possible, its not blocking me at the moment
21:21:33  <dominictarr>ralphtheninja: what is the "too depressing" thing that he doesn't talk about?
21:21:42  <dominictarr>do you have more info on that?
21:22:05  <trevnorris>mbalho: ok cool. yeah, there are definitely ways to do what you want. probably a much simpler one in JS.
21:22:50  <trevnorris>mbalho: but tell you the truth, been looking for a reason to write a module to do what I explained on the native layer. so if you need it, let me know. :)
21:23:06  <trevnorris>usually just don't get around to something unless I really need it.
21:27:48  <ralphtheninja>dominictarr: have to walk jazon, bbiab
21:28:14  <dominictarr>ralphtheninja: tell jazon I say hi
21:32:59  * AvianFluquit (Remote host closed the connection)
21:37:16  <dominictarr>ralphtheninja: that talk is crazy!
21:38:25  * anoemiquit (Quit: anoemi)
21:43:20  * venjoined
22:02:30  * LOUDBOTquit (Ping timeout: 264 seconds)
22:03:06  <ralphtheninja>dominictarr: it's awesome :)
22:04:00  * timoxleyjoined
22:05:44  <mbalho>dominictarr: with the redis protocol i'm trying to figure out if its okay if i have newlines inside my data or if i have to escape them
22:05:58  <mbalho>dominictarr: cause if its framed then it shouldnt matter if i have extra newlines
22:06:15  <mbalho>dominictarr: and i'd really like the data to be the actual data and not have to do a post-process step to unescape
22:06:41  <substack>mbalho: https://gist.github.com/mk30/6757010
22:06:51  <substack>concat-stream bug?
22:07:18  <substack>or rather, bops bug
22:07:19  <mbalho>substack: its node >= 0.8
22:07:37  <ralphtheninja>dominictarr: I don't know what the law he's talking about, I'll ask around
22:07:45  <substack>mbalho: aha, right
22:07:57  * LOUDBOTjoined
22:08:21  * timoxleyquit (Ping timeout: 245 seconds)
22:08:28  * dominictarrquit (Ping timeout: 240 seconds)
22:14:34  * dominictarrjoined
22:16:39  * jcrugzzquit (Ping timeout: 252 seconds)
22:19:29  * jergasonquit (Quit: jergason)
22:25:52  <ralphtheninja>dominictarr: here it is http://en.m.wikipedia.org/wiki/Foreign_Account_Tax_Compliance_Act
22:26:15  <ralphtheninja>dominictarr: kind of cool that he mentions copters too
22:32:56  * thlorenzquit (Remote host closed the connection)
22:34:05  <mbalho>dominictarr: my feeling is that the redis-protocol lets you put \r\n inside your data, it just uses the \r\n as a operation delimiter to make it human readable
22:38:23  <mbalho>dominictarr: but if you have multiline data in a redis protocol stream e.g. *1\r\n$4\r\nh\r\ni
22:38:29  <mbalho>dominictarr: it will show up as
22:38:31  <mbalho>*1
22:38:34  <mbalho>$4
22:38:36  <mbalho>h
22:38:36  <mbalho>i
22:38:53  * ferossjoined
22:39:01  <mbalho>dominictarr: but h\r\ni is actually one 'line', and the parser will still parse it correctly
22:39:12  <mbalho>dominictarr: it will just get rendered with newlines when printed out, but i think thats okay
22:39:20  <mbalho>dominictarr: i cant find any documentation about this online for some reason
22:43:11  * dominictarrquit (Ping timeout: 260 seconds)
22:45:09  * dominictarrjoined
22:50:59  * AvianFlujoined
23:04:47  * timoxleyjoined
23:06:40  * AvianFluquit (Remote host closed the connection)
23:07:49  * AvianFlujoined
23:09:19  * timoxleyquit (Ping timeout: 260 seconds)
23:19:44  <ralphtheninja>dominictarr: we could start building a trade net on peerconf :)
23:22:17  * dguttmanquit (Quit: dguttman)
23:25:07  * anoemijoined
23:27:34  * stagasquit (Read error: Connection reset by peer)
23:32:31  * ferossquit (Quit: feross)
23:33:21  * jxsonjoined
23:37:44  * jxsonquit (Ping timeout: 256 seconds)
23:45:00  * st_lukejoined
23:45:26  * dominictarrquit (Ping timeout: 245 seconds)
23:47:18  * dominictarrjoined
23:47:26  <st_luke>Domenic_: what happened lately, so many broken things
23:47:57  <Domenic_>st_luke: I know, really strange. Drive by Isaac commits before vacation, maybe? Not sure.
23:49:45  <st_luke>https://github.com/isaacs/npm/commit/2389525a1df6d20e780cca5887f1b73ff583ce8d#commitcomment-4207705 passes for me on unix
23:52:03  <st_luke>Domenic_: im actually kinda glad that the isaacbw guy in nodejs irc keeps mental notes of issues people have
23:58:43  * mikolalysenkojoined