00:11:35  * mageemooneyjoined
00:12:40  * thlorenzjoined
00:14:07  * wolfeidaujoined
00:14:53  * wolfeidauquit (Remote host closed the connection)
00:15:22  * wolfeidaujoined
00:15:23  * mageemooneyquit (Remote host closed the connection)
00:16:05  * mageemooneyjoined
00:18:10  * ednapiranhaquit (Quit: Leaving...)
00:20:18  * mageemooneyquit (Ping timeout: 245 seconds)
00:21:10  * jjmalinajoined
00:26:57  * thlorenzquit (Ping timeout: 276 seconds)
00:44:45  * thlorenzjoined
00:46:53  * thlorenzquit (Remote host closed the connection)
01:02:56  * ashihjoined
01:05:56  * paulfryzelquit (Remote host closed the connection)
01:06:34  * paulfryzeljoined
01:07:37  * mcavagequit (Remote host closed the connection)
01:08:03  * mcavagejoined
01:10:43  * paulfryzelquit (Ping timeout: 252 seconds)
01:12:52  * mcavagequit (Ping timeout: 265 seconds)
01:12:58  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
01:16:37  * mageemooneyjoined
01:21:47  * mageemooneyquit (Ping timeout: 258 seconds)
01:29:24  * ashihjoined
01:32:14  * ashih_joined
01:32:30  * ramitosquit (Read error: Connection reset by peer)
01:33:28  * jerrysvquit (Remote host closed the connection)
01:33:38  * ashihquit (Ping timeout: 245 seconds)
01:39:09  * ednapiranhajoined
01:40:12  * ashihjoined
01:41:01  * ashih_quit (Ping timeout: 250 seconds)
01:51:29  * fritzyjoined
01:53:18  * ednapiranhaquit (Remote host closed the connection)
01:53:44  * ednapiranhajoined
02:07:01  * ednapiranhaquit (Ping timeout: 258 seconds)
02:07:08  * paulfryz_joined
02:11:04  * mageemooneyjoined
02:15:37  * mageemooneyquit (Ping timeout: 252 seconds)
02:16:39  * mageemooneyjoined
02:19:39  * paulfryz_quit (Ping timeout: 252 seconds)
02:21:07  * mageemooneyquit (Ping timeout: 252 seconds)
02:39:55  * ashih_joined
02:39:55  * ashihquit (Read error: Connection reset by peer)
02:43:12  * thlorenzjoined
02:43:36  * ednapiranhajoined
02:43:43  * mcavagejoined
02:48:02  * mcavagequit (Ping timeout: 258 seconds)
02:57:46  * ednapiranhaquit (Ping timeout: 276 seconds)
02:58:17  <levelbot>[npm] [email protected] <http://npm.im/dat>: real-time replication and versioning for large tabular data sets (@maxogden)
02:59:43  * paulfryz_joined
03:04:01  * paulfryz_quit (Ping timeout: 252 seconds)
03:10:57  * mageemooneyjoined
03:12:59  * mageemoo_joined
03:16:07  * mageemooneyquit (Ping timeout: 265 seconds)
03:16:33  * mageemoo_quit (Read error: Connection reset by peer)
03:19:52  * tphummelquit (Quit: tphummel)
03:22:34  * tphummeljoined
03:23:19  * ednapiranhajoined
03:27:54  * ednapiranhaquit (Ping timeout: 258 seconds)
03:29:16  * tphummelquit (Quit: tphummel)
03:29:17  * jerrysvjoined
03:30:01  * jerrysvquit (Remote host closed the connection)
03:30:08  * ednapiranhajoined
03:33:44  * tphummeljoined
03:34:23  * thlorenzquit (Remote host closed the connection)
03:34:56  * thlorenzjoined
03:35:23  * tphummelquit (Client Quit)
03:39:24  * thlorenzquit (Ping timeout: 258 seconds)
03:42:28  * ednapiranhaquit (Ping timeout: 258 seconds)
03:53:45  <levelbot>[npm] [email protected] <http://npm.im/dat>: real-time replication and versioning for large tabular data sets (@maxogden)
03:55:09  <fritzy>dat is interesting
03:58:25  <ogd>fritzy: working feverishly on it, we'll be doing a proper 'release' soon that will hopefully make a lot more sense
04:03:02  <fritzy>ogd: Sounds good. I'm curious
04:04:10  * saibotvisadjoined
04:10:49  * mageemooneyjoined
04:15:30  * mageemooneyquit (Ping timeout: 240 seconds)
04:16:39  * mageemooneyjoined
04:20:51  * mageemooneyquit (Ping timeout: 240 seconds)
04:27:56  * ednapiranhajoined
04:29:34  * jjmalinaquit (Quit: Textual IRC Client: www.textualapp.com)
04:32:27  * ednapiranhaquit (Ping timeout: 240 seconds)
04:43:35  * mikealquit (Quit: Leaving.)
04:48:11  * calvinmetcalfquit (Quit: Connection closed for inactivity)
05:07:09  * saibotvisadquit (Quit: Leaving.)
05:08:26  * fritzyquit (Remote host closed the connection)
05:16:03  * brianloveswordsjoined
05:18:07  * ednapiranhajoined
05:18:58  * wolfeidauquit (Remote host closed the connection)
05:28:39  * Sorellaquit (Quit: It is tiem!)
05:58:49  * brianloveswordsquit (Quit: Computer has gone to sleep.)
06:01:18  * ashihjoined
06:02:00  * paulfryz_joined
06:02:22  * ashih_quit (Ping timeout: 276 seconds)
06:06:37  * paulfryz_quit (Ping timeout: 252 seconds)
06:08:24  * mikealjoined
06:16:56  * ednapira_joined
06:17:10  * wolfeidaujoined
06:17:10  * ednapiranhaquit (Read error: Connection reset by peer)
06:17:17  * ednapira_changed nick to ednapiranha
06:23:11  * ednapiranhaquit (Quit: Leaving...)
06:34:15  * fritzyjoined
06:44:45  * fritzyquit
06:56:30  * ashih_joined
06:58:44  * ashihquit (Ping timeout: 258 seconds)
07:02:51  * paulfryz_joined
07:06:58  * ashih_quit (Ping timeout: 245 seconds)
07:07:07  * paulfryz_quit (Ping timeout: 252 seconds)
07:08:58  * ashihjoined
07:55:09  * ashihquit (Ping timeout: 252 seconds)
07:55:57  * ashihjoined
08:03:34  * paulfryz_joined
08:07:59  * paulfryz_quit (Ping timeout: 252 seconds)
08:17:53  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
09:04:21  * paulfryz_joined
09:08:51  * paulfryz_quit (Ping timeout: 252 seconds)
09:40:40  * binocarlosquit (Read error: Connection reset by peer)
09:40:50  * binocarlosjoined
10:05:05  * paulfryz_joined
10:09:21  * paulfryz_quit (Ping timeout: 252 seconds)
10:43:46  * no9joined
10:59:33  * no9quit (Quit: This computer has gone to sleep)
11:05:57  * paulfryz_joined
11:10:18  * paulfryz_quit (Ping timeout: 245 seconds)
11:41:36  * calvinmetcalfjoined
12:06:43  * paulfryz_joined
12:07:07  * mafintosh_joined
12:10:29  * ehd_joined
12:11:05  * paulfryz_quit (Ping timeout: 252 seconds)
12:13:42  * no9joined
12:14:52  * mafintoshquit (*.net *.split)
12:14:53  * ogdquit (*.net *.split)
12:14:53  * ehdquit (*.net *.split)
12:15:21  * mafintosh_changed nick to mafintosh
12:15:32  * ehd_changed nick to ehd
12:20:21  * ogdjoined
12:25:29  * no9quit (Quit: This computer has gone to sleep)
13:07:25  * paulfryz_joined
13:11:57  * paulfryz_quit (Ping timeout: 252 seconds)
13:21:25  * ralphtheninjajoined
13:43:52  * thlorenzjoined
13:45:34  * thlorenzquit (Remote host closed the connection)
13:48:11  * calvinmetcalfquit (Quit: Connection closed for inactivity)
14:03:19  * thlorenzjoined
14:08:11  * paulfryz_joined
14:09:09  <ralphtheninja>testing 1 2 3
14:12:40  * paulfryz_quit (Ping timeout: 258 seconds)
14:16:45  * thlorenzquit (Ping timeout: 250 seconds)
14:27:07  * mageemooneyjoined
14:28:02  * mageemooneyquit (Remote host closed the connection)
14:29:07  * mageemooneyjoined
14:31:31  * Sorellajoined
14:51:28  * calvinmetcalfjoined
14:55:21  * no9joined
15:08:55  * paulfryzeljoined
15:13:19  * paulfryzelquit (Ping timeout: 252 seconds)
15:14:49  * jerrysvjoined
15:16:40  * thlorenzjoined
15:23:25  * brianloveswordsjoined
15:24:13  * mcavagejoined
15:29:17  * no9quit (Quit: This computer has gone to sleep)
15:30:06  * mcavagequit (Remote host closed the connection)
15:30:22  * sygijoined
15:46:30  * no9joined
15:52:01  * mcavagejoined
15:56:04  * mcavagequit (Remote host closed the connection)
15:59:08  * ashihjoined
16:00:33  * ashih_joined
16:04:15  * ashihquit (Ping timeout: 276 seconds)
16:09:42  * paulfryzeljoined
16:14:39  * paulfryzelquit (Ping timeout: 276 seconds)
16:17:30  * tphummeljoined
16:25:08  * mcavagejoined
16:29:25  * mcavage_joined
16:29:54  * mcavagequit (Ping timeout: 258 seconds)
16:35:31  * tphummelquit (Quit: tphummel)
16:36:13  * mcavage_quit (Remote host closed the connection)
16:43:13  * ednapiranhajoined
16:49:00  * no9quit (Quit: Leaving)
16:49:08  * mikealquit (Quit: Leaving.)
17:10:28  * paulfryzeljoined
17:13:25  * thlorenzquit (Remote host closed the connection)
17:15:03  * paulfryzelquit (Ping timeout: 252 seconds)
17:23:38  * ashih_quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
17:31:50  * mageemooneyquit (Remote host closed the connection)
17:32:25  * mageemooneyjoined
17:36:42  * mageemooneyquit (Ping timeout: 240 seconds)
17:42:03  * tphummeljoined
18:07:11  * ednapiranhaquit (Quit: Leaving...)
18:07:56  * thlorenzjoined
18:11:21  * paulfryzeljoined
18:12:22  * sygiquit (Quit: Connection closed for inactivity)
18:15:55  * paulfryzelquit (Ping timeout: 252 seconds)
18:17:48  * kenan|afkchanged nick to kenansulayman
18:23:08  * tphummelquit (Quit: tphummel)
18:50:20  * mikealjoined
18:50:52  * ogdchanged nick to mbalho
18:52:55  * jerrysv_joined
18:54:32  * mikealquit (Ping timeout: 250 seconds)
18:55:05  * mbalhochanged nick to ogd
18:56:20  * jerrysvquit (Ping timeout: 258 seconds)
19:11:58  * paulfryzeljoined
19:16:25  * paulfryzelquit (Ping timeout: 252 seconds)
20:12:56  * paulfryzeljoined
20:17:17  * paulfryzelquit (Ping timeout: 252 seconds)
20:18:11  * calvinmetcalfquit (Quit: Connection closed for inactivity)
20:23:36  * No9joined
20:34:03  * dguttmanjoined
20:50:10  * mikealjoined
20:54:48  * mikealquit (Ping timeout: 276 seconds)
20:58:34  * ashihjoined
21:11:44  * thlorenzquit (Remote host closed the connection)
21:13:31  * paulfryzeljoined
21:18:09  * paulfryzelquit (Ping timeout: 252 seconds)
21:38:01  * jerrysv_quit (Remote host closed the connection)
21:39:07  * brianloveswordsquit (Quit: Computer has gone to sleep.)
21:41:12  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
21:47:42  * briancjoined
21:47:47  <brianc>hey I got a queeestion
21:48:03  <brianc>it's not about leveldb really but asking in here because the node.js irc room is kinda hard to get anyone to say anything useful anymore
21:48:18  <brianc>I have a 500 megabyte json file
21:48:23  <brianc>I need to load this whole file into memory in node
21:48:37  <brianc>but it starts to cause major GC thrashing when it gets near the heap limit
21:48:45  <brianc>is there a module you know of to store a bunch of stuff outside the heap?
21:48:59  <brianc>i literally just need an array of json objects not in the v8 heap. they are never collected for the life of the entire application
21:49:46  <brianc>but i would like to be able to itterate throug this array as fast as possible
21:49:57  <brianc>in memory database kinda thing
21:56:33  <ogd>brianc: why do you need all 500mb of json in memory
21:57:17  <brianc>because I need to compare 1 object to every object in the data set (2 million items) in under 1 second
21:58:06  <ogd>brianc: what exactly are you comparing?
21:58:13  <brianc>if i only load about 800k items it does the comps and sorting in about 300 ms
21:58:22  <brianc>if I load 1.3MM items it takes 50 seconds to do the same thing
21:58:55  <brianc>comparing the basis item to all other items in the data set across a wide variety of dimensions and scoring the similarity. then sorting the scores and returning the top 100 most similar items
21:59:33  * ashihjoined
21:59:52  <ogd>what does 'wide variety of dimensions' mean exactly? e.g. what does your data look like what what subset of each object do you need to access
22:00:32  <brianc>sorry i'll be less vague
22:00:51  <ogd>mostly im wondering if your objects are 1 level deep or super nested tree-like
22:01:32  <brianc>i have a json object that is like this: {age: 100, height: 50, width: 30, price: 35, latitude: 31.310101, longitude: 33.301013} about 30 numeric properties
22:01:36  <brianc>they are all 1 level deep
22:02:09  <brianc>and then I compare a basis to all the other objects & take weighted variance of how different each different dimension is, and those variances all add up to a final score
22:02:37  <brianc>so I can say "of the 2MM items, here are the 100 most like the one you're looking at now"
22:02:39  <ogd>brianc: gotcha. do you use the same attributes on every object? and do you use all attributes on all objects or just a subset?
22:03:35  <brianc>the data is 100% uniform - all objects compared are the same. and I use all the attributes of each object in a 'worst case' scenario. depending on what filters are enabled some will get filtered out of the compare before all compares run in some cases
22:04:07  <brianc>I haven't spent a lot of time making the code fast and the results are already blowing my mind so I'm excited. Only hurdle I can see is the heap limit.
22:04:22  <ogd>brianc: ok last question, do you know ahead of time what the maximum float/int precision is on your data?
22:04:44  <brianc>doesn't have to be very precise
22:04:58  <brianc>the fact that I don't even know the answer to that question probably a good indication of the precision I need. :p
22:05:01  <ogd>hah
22:05:31  <ogd>brianc: ok so check out https://github.com/mikolalysenko/ndarray
22:05:52  <ogd>brianc: the tl;dr is that if you were to go and make 2 million typed arrays it might work... but i doubt it
22:05:55  <ogd>worth a shot though
22:06:11  <brianc>are typed arrays still on the heap?
22:06:15  <ogd>nope
22:06:19  * briancdances
22:06:24  <ogd>neither are Buffers
22:06:30  <ogd>but
22:06:41  <ogd>say your objects have 50 properties
22:07:00  <ogd>you would create a typed array of length (50 * 2 million)
22:07:28  <ogd>and then do require('ndarray')(yourTypedArray, [20, 2 million])
22:07:29  <ogd>or something like that
22:07:36  <brianc>nice
22:07:37  <ogd>and that will return a virtual 2d table basically
22:07:42  <ogd>that is 50 wide and 2 million tall
22:07:50  <ogd>then you can fill it up to your hearts desire with data
22:07:53  <brianc>going to be fun to maintain the code once it's all array indices instead of property names
22:08:23  <ogd>shouldnt be too bad, just adds an extra indexOf check
22:08:26  <ogd>also
22:08:37  <ogd>50 * 2 million is a lot
22:08:41  <brianc>yeah
22:08:52  <ogd>but at least this will let you use all your RAM
22:08:56  <ogd>also
22:09:04  <ogd>you'll wanna parse the json one at a time and fill in the ndarray grid
22:09:14  <brianc>we've tried a lot of different ideas. currently it's implemented in C...i'd like to not maintain C since this is quite a bit of important business logic that changes on a somewhat regular basis
22:09:25  <brianc>yeah parsing the json is already done w/ JSONStream
22:09:50  <brianc>i could also load it directly from postgres into memory in a streaming way, so I think I got that covered
22:11:10  <brianc>one other question...if tyepdarrays aren't on the heap and you can have as many as you have ram...could I just make a normal javascript array full of typed arrays?
22:11:11  <ogd>so if you did int8arrays it should only take up ~768mb of memory if i did my math correctly
22:11:53  <ogd>the cost of instantiating typed arrays is rather high, read the first few links on the ndarray readme + the blog post
22:12:04  <brianc>okay cool
22:12:06  <ogd>i could be wrong though
22:12:12  <brianc>i load the data once at process start and startup time is not a factor
22:12:41  <brianc>my product guy is basically like "I want everything in our entire data set analyzed in all these crazy ways in under a second" so however I can make that happen we win!
22:12:57  <ogd>hehe
22:13:08  <ogd>i mean, you'll wanna avoid creating JS objects here
22:13:19  <ogd>the fewer typed arrays/arrays the better
22:13:27  <brianc>okay cool
22:13:47  <ogd>also sounds like there are some generic pieces of functiionality you're building, you should write some modules!
22:14:13  <brianc>hell yeah
22:14:14  * paulfryzeljoined
22:14:16  <brianc>are you max ogden?
22:14:18  <ogd>yep
22:14:27  <brianc>awesome - I met you briefly at node-conf last year
22:14:30  <ogd>oh nice
22:14:33  <brianc>your talk was one of my favs
22:14:39  <ogd>the postgres stream code sounds cool
22:14:41  <brianc>"wooo!"
22:14:44  <ogd>haha
22:14:56  <brianc>https://github.com/brianc/node-pg-query-stream
22:14:59  <ogd>oh nice
22:15:41  <ogd>that looks useful
22:15:45  <brianc>yeah I've been doing a lot of ETL and postgres + streams = fast, fun ETL with node
22:16:28  <ogd>brianc: oooooh keep an eye on my project http://dat-data.com/, it's basically my attempt to reinvent ETL w/ node modules
22:16:47  <brianc>been watching it. :)
22:16:51  <ogd>w00t
22:17:21  <ogd>working on https://github.com/jbenet/transformer and a stable release of dat itself now, shooting for next month
22:17:39  <ogd>gonna try and make streaming data modules for all the things
22:17:45  <brianc>cannot wait until we can start ingesting data this way
22:17:59  <ogd>are you comign to nodeconf this year?
22:18:07  <brianc>Not this year. :(
22:18:10  <ogd>ahh bummer
22:18:23  <brianc>Moving across the country in a few weeks and working like crazy hours trying to get my startup launched
22:18:27  <brianc>same old story. :p
22:18:30  <ogd>haha
22:18:33  <brianc>it is a bummer - last year was awesome
22:18:39  * paulfryzelquit (Ping timeout: 252 seconds)
22:19:11  <ogd>brianc: is it https://www.reonomy.com/ ?
22:19:12  <brianc>thanks a lot for your help today - and all the stuff you've done for node. :)
22:19:13  <brianc>yeah
22:19:21  <ogd>nide
22:19:23  <ogd>nice*
22:19:31  <ogd>if you make hexagon stickers make sure to print them 2" wide
22:19:41  <brianc>hahaha okay
22:19:52  <ogd>well 2" for the longest diagonal
22:19:57  <ogd>its like 2" x 1.75" or something
22:20:02  <brianc>will they perfectly match node stickers then?
22:20:22  <ogd>https://twitter.com/dybskiy/status/454348392856944640
22:20:49  <ogd>gotta run, have fun with ndarray
22:21:06  <brianc>thanks! :)
22:23:02  * dguttmanquit (Quit: dguttman)
22:31:55  * ashihquit (Quit: My MacBook has gone to sleep. ZZZzzz…)
23:11:47  <levelbot>[npm] [email protected] <http://npm.im/dat>: real-time replication and versioning for large tabular data sets (@maxogden)
23:15:03  * paulfryz_joined
23:19:31  * paulfryz_quit (Ping timeout: 252 seconds)
23:46:08  * ednapiranhajoined
23:46:10  * ednapiranhaquit (Remote host closed the connection)
23:46:17  * ednapira_joined
23:46:39  * ednapira_changed nick to ednapiranha