00:05:56  * jcrugzzjoined
00:19:23  * thlorenzjoined
00:48:58  * esundahljoined
00:51:26  * insertcoffeequit (Ping timeout: 240 seconds)
01:28:28  * kenansul_joined
01:31:40  * kenansulaymanquit (Ping timeout: 264 seconds)
01:45:35  * kenansul_quit (Remote host closed the connection)
02:06:36  * jmartinsjoined
02:33:20  * jmartinsquit (Quit: Konversation terminated!)
02:44:05  * jmartinsjoined
02:53:31  * jmartinsquit (Read error: Connection reset by peer)
03:03:36  * esundahlquit (Remote host closed the connection)
03:11:09  * tarrudajoined
03:11:48  * gwenbelljoined
03:17:08  * jcrugzzquit (Ping timeout: 240 seconds)
03:30:22  * esundahljoined
03:37:08  * timoxleyjoined
03:45:29  * jcrugzzjoined
03:47:56  * timoxleyquit (Remote host closed the connection)
04:14:17  * timoxleyjoined
04:24:33  * tarrudaquit (Quit: leaving)
04:26:51  * thlorenzquit (Remote host closed the connection)
04:30:00  * eugenewarechanged nick to zz_eugeneware
04:42:00  * jcrugzzquit (Ping timeout: 253 seconds)
04:44:21  * gwenbellquit (Ping timeout: 245 seconds)
04:51:40  * jcrugzzjoined
04:56:14  * jcrugzzquit (Ping timeout: 240 seconds)
05:02:00  * jcrugzzjoined
05:07:18  * jcrugzzquit (Ping timeout: 265 seconds)
05:07:54  * jcrugzzjoined
05:09:06  * zz_eugenewarechanged nick to eugeneware
05:11:28  * DTrejoquit (Remote host closed the connection)
05:17:08  * jcrugzzquit (Ping timeout: 240 seconds)
05:19:30  * eugenewarechanged nick to zz_eugeneware
05:37:57  * esundahlquit (Ping timeout: 272 seconds)
05:43:33  * mikealquit (Quit: Leaving.)
05:44:29  * mikealjoined
06:14:56  * zz_eugenewarechanged nick to eugeneware
06:25:28  * jcrugzzjoined
06:54:56  * thlorenzjoined
07:03:02  * thlorenzquit (Ping timeout: 240 seconds)
07:29:09  * thlorenzjoined
07:35:01  * timoxleyquit (Remote host closed the connection)
07:37:10  * thlorenzquit (Ping timeout: 246 seconds)
07:57:44  * dominictarrjoined
08:24:07  * thlorenzjoined
08:25:47  * dominictarrquit (Ping timeout: 272 seconds)
08:29:13  * thlorenzquit (Ping timeout: 272 seconds)
08:34:03  * thlorenzjoined
08:35:31  * timoxleyjoined
08:39:05  * thlorenzquit (Ping timeout: 272 seconds)
09:10:31  * fb55joined
09:15:00  * timoxleyquit (Remote host closed the connection)
09:15:27  * fb55quit (Ping timeout: 272 seconds)
09:31:20  * DTrejojoined
09:34:44  * thlorenzjoined
09:39:02  * thlorenzquit (Ping timeout: 240 seconds)
10:06:26  * nathan7part
10:35:22  * thlorenzjoined
10:39:27  * thlorenzquit (Ping timeout: 240 seconds)
11:04:34  * DTrejoquit (Remote host closed the connection)
11:05:01  * DTrejojoined
11:05:52  * DTrejoquit (Read error: Connection reset by peer)
11:09:53  * dscapepart
11:33:28  * DTrejojoined
11:33:33  * DTrejoquit (Read error: Connection reset by peer)
11:35:59  * thlorenzjoined
11:40:38  * thlorenzquit (Ping timeout: 256 seconds)
11:49:36  * tarrudajoined
12:11:49  * kenansulaymanjoined
12:15:04  * DTrejojoined
12:15:42  * DTrejoquit (Read error: Connection reset by peer)
12:27:40  * jcrugzzquit (Ping timeout: 246 seconds)
12:54:58  * jcrugzzjoined
13:03:53  * jcrugzzquit (Ping timeout: 265 seconds)
13:35:08  <levelbot>[npm] [email protected] <http://npm.im/contextdb>: Use json-context with leveldb. Contexts are automatically generated from matchers, and provides ability to watch matchers for realtime notifications. (@mmckegg)
13:37:14  * thlorenzjoined
13:41:31  * thlorenzquit (Ping timeout: 246 seconds)
13:50:38  <levelbot>[npm] [email protected] <http://npm.im/level-stay>: Persistence for scuttlebutts based on LevelDB (@juliangruber)
13:52:58  * jcrugzzjoined
13:55:59  * DTrejojoined
13:56:17  * DTrejoquit (Read error: Connection reset by peer)
14:05:33  * DTrejojoined
14:05:51  * DTrejoquit (Read error: Connection reset by peer)
14:18:33  * tarrudaquit (Ping timeout: 272 seconds)
14:27:44  * DTrejojoined
14:27:50  * DTrejoquit (Read error: Connection reset by peer)
14:32:07  * jcrugzzquit (Ping timeout: 272 seconds)
14:54:18  * DTrejojoined
14:54:31  * DTrejoquit (Read error: Connection reset by peer)
14:56:06  * DTrejojoined
14:56:15  * DTrejoquit (Read error: Connection reset by peer)
14:56:36  * DTrejojoined
14:56:41  * DTrejoquit (Read error: Connection reset by peer)
14:59:45  * mikealquit (Quit: Leaving.)
15:08:49  * thlorenzjoined
15:08:50  * thlorenzquit (Remote host closed the connection)
15:11:54  * thlorenzjoined
15:11:54  * thlorenzquit (Remote host closed the connection)
15:12:25  * thlorenzjoined
15:25:08  <levelbot>[npm] [email protected] <http://npm.im/meatspace-leveldb>: Decentralized micro[b]logging with leveldb (@ednapiranha)
15:25:11  * mikealjoined
15:49:45  <kenansulayman>yoyo guys
15:50:10  <kenansulayman>rvagg rescrv ogd Had a test run with mmalecki at Sophia
15:50:14  <kenansulayman>Check this out
15:50:14  <kenansulayman>https://github.com/mmalecki/node-sophia/issues/3
15:53:08  <kenansulayman>Batches etc is to come
15:53:38  <levelbot>[npm] [email protected] <http://npm.im/lev>: commandline and REPL access for leveldb (@hij1nx, @juliangruber)
15:53:39  <kenansulayman>Yet we got put get del running (and fixed Sophia not compiling on OSX)
15:58:53  * fb55joined
16:05:12  * mikealquit (Quit: Leaving.)
16:06:23  * mikealjoined
16:18:07  * mikealquit (Quit: Leaving.)
16:31:37  * jcrugzzjoined
16:31:42  * fb55quit (Read error: Connection reset by peer)
16:48:53  * mikealjoined
17:09:27  * jcrugzzquit (Ping timeout: 252 seconds)
17:12:38  <levelbot>[npm] [email protected] <http://npm.im/lev>: commandline and REPL access for leveldb (@hij1nx, @juliangruber)
17:25:48  * thlorenzquit (Remote host closed the connection)
17:40:04  <rescrv>kenansulayman: how about a summary? What's the workload? What's the setup? Is there already data in the database? What insight should I take away?
17:42:32  <kenansulayman>rescrv this is not finetuned, no special adjustments or flags. Just an empty database. I write a million entries from a csv which I previously read into memory so that there is no high fluctuation because of disk-usage issues (I'm on a HDD)
17:43:52  <kenansulayman>They keys are Strings generated by ~~(Math.random()*1E17) and split in the read process
17:44:57  <kenansulayman>Since batch etc isn't implemented in the driver yet the quite low level put to write is used
17:45:37  <rescrv>kenansulayman: I'm still not sure what I should be learning from the issue you linked.
17:45:57  <rescrv>I see that sophia takes 6s, HyperLevelDB takes 15s, and that you love makefiles
17:46:35  <rescrv>but I don't see any description of what it is you're doing, why it is a valid comparison, or what I should learn from the results
17:46:42  <kenansulayman>rescrv I don't know what you'd "learn" it's just a comparision in speed for cases that apply for us in production (we wouldn't actually batch since we write per request of a user)
17:47:05  <rescrv>To do that, I have to download the CSV, which still doesn't tell me much as it's just random strings.
17:47:41  <kenansulayman>rescrv Why are you so negative? I did not want to offend if I did.
17:48:53  <rescrv>kenansulayman: then say that. That tells me something I didn't know before. Something like, "In production, we often have to do X. X is characterized by Y. This benchmark mimics Y by measuring Z. As you can see, ..."
17:49:46  <rescrv>I'm not trying to be negative. Maybe I've just had too much coffee/too little sleep. I also tend to get frustrated by benchmarks that show raw numbers that reflect on (part of) my work, without explaining anything.
17:50:03  * thlorenzjoined
17:50:06  <kenansulayman>rescrv It
17:50:15  <rescrv>one little benchmark saying something is bad/broken, and it's many hours of headaches down the line defending it.
17:50:47  <rescrv>I appreciate that you took time to benchmark hyperleveldb, and perhaps am being a little rude in trying to understand what's going on.
17:51:15  <kenansulayman>rescrv It is actually that it took quite some time to get right, so it's rather to show off the work since rvagg talked about it being a possible new level backend
17:52:47  <kenansulayman>rescrv and I didn't chose hyperlevel because I would like to say "this" or "that" is better; actually hyperlevel and lmdb is what we use / used in production; that is I just tried to (I admit it's quite blurry) get a grasp on Sophia's functions and performance
17:53:03  <rescrv>kenansulayman: understood
17:53:25  <rescrv>I overreacted.
17:53:41  <rescrv>out of curiosity, what'd you use HyperLevelDB in production for/where'd you use it?
17:55:33  <kenansulayman>At several places: primarily user avatar cache andsession storage
17:56:23  <kenansulayman>We used it previously as backend for the main-database of for all the users (with replication) but switched to lmdb because it allows clustering of the process
17:56:40  <kenansulayman>used it until recently*
17:58:00  <kenansulayman>(clustering = just a new instance of the server using the same database; we usually spawn as many instances of the server as there are cores)
17:58:19  <rescrv>my main concern with lmdb is that it manages its own heap. Which is fine standalone, but the changes coming down the pipe in HyperDex leverage LevelDB's structure in huge ways, to allow quick backup and server-server repair. Sophia/LMDB cannot match that
18:00:04  * thlorenzquit (Remote host closed the connection)
18:00:12  <kenansulayman>Could you elaborate on why it's a concern of yours?
18:02:08  <rescrv>well, it's mainly a concern in clustered/replicated production environments. Say I want to take a backup of a server. Stop the server, backup, start the server. Except if you have a lot of data, that backup is expensive and translates into downtime.
18:02:24  <rescrv>We modified LevelDB to do it nearly instantly, leveraging FS-level hardlinks.
18:02:58  <rescrv>AFAIK, you cannot do that with things that manage their own heap as one file. You need invasive modification to enable that.
18:03:48  <rescrv>So, in HyperLevelDB, I call "LiveBackup" and it's done. It takes literally milliseconds, even if you have terabytes of data. You can then do differential backup, rsync'ing from the host, only that data that changed since your last backup.
18:04:15  <kenansulayman>hm. What stops us from just copying the directory while load-balancing the data to another server?
18:04:25  <rescrv>Similar tricks enable us to transfer a small subset of data to re-sync replicas, rather than having to scan everything.
18:04:40  <kenansulayman>Ah! Is that possible over the node hyperleveldb binding?
18:04:57  <rescrv>I don't know your arch, but for us that means that you now have to take something offline and keep it offline while copying.
18:05:28  <rescrv>For HyperDex, we can pause the entire cluster, backup every node at a consistent snapshot, and unpause the writes, to give a fully 100% consistent backup with just a few seconds of downtime.
18:05:52  <rescrv>the live backup? I don't know if it's exposed over Node. I haven't looked.
18:06:11  <brycebaril>I think LiveBackup was exposed (via ogd) but I haven't tried it yet.
18:06:12  <kenansulayman>That's to HyperDex. I kept nagging you about a node binding to that ;)
18:07:22  <kenansulayman>brycebaril Have you got a link to that? :)
18:07:23  <brycebaril>rescrv: do you pause the cluster for the consistency? That way no need to track missed operation deltas between LiveBackup init & completion?
18:08:21  <rescrv>brycebaril: exactly. It's (theoretically) also possible to take a sloppy backup that you'd then have to replay against an old version, but I've got enough to do already.
18:08:28  <brycebaril>kenansulayman: : look at the commit log for the node-LevelDOWN HyperLevelDB branch
18:08:53  <rescrv>kenansulayman: I'm illustrating how it's useful. If you shard across different instances in your app, you'd be able to take the same approach to backup your dat.
18:10:07  <brycebaril>rescrv: Yeah, in a clustered environment I agree that's the easiest way to get consistency. It'll be interesting to see what Redis does for it's cluster. Right now it's equivalent of LiveBackup that it uses for replication (SYNC) tracks the delta and sends that. But that's a single-master environment.
18:12:36  <rescrv>brycebaril: I think they are poking around in the dark, looking for a solution. In an ideal world, you'd use this: http://ece842.com/S13/readings/chandy85.pdf to take the snapshot.
18:12:43  <kenansulayman>rescrv I'd love to get started with HyperDex since it's hard to maintain our own replication / master-slave code while doing the other work, too.
18:12:49  <rescrv>the complexity of getting that right in the implementation
18:13:32  <rescrv>it would keep me up at night, and I'd much rather give people the ability to either pause the cluster, or have to restore after failure.
18:14:09  <rescrv>kenansulayman: I cannot wait to get htis release done so I can get bindings ready, but there's a few other surprises we're getting ready first.
18:15:05  <kenansulayman>rescrv I am really curious. After that Google employee told me about the pricing of their K/V and that every transaction costs actually 3 writes I cringed
18:15:59  <rescrv>can you fill me in on a little more? What system? Spanner?
18:16:33  <kenansulayman>Spanner?
18:17:42  <rescrv>so spanner's got a neat design, and they don't care about cost because they can run more servers to overcome it
18:18:17  <brycebaril>rescrv: this paper looks cool. Long shot, but do you know of similar work for write-only (i.e. no updates/deletes) databases? my assumption is that vastly simplifies the problemset
18:18:33  <rescrv>the three writes aren't the problem as much as having to do a wide-area round trip through paxos to commit
18:19:44  <kenansulayman>rescrv We recently been to Google HQ berlin to discuss our adwords campaign and we met a group of cloud engineers there. We talked about our database to keep up with realtime collaboration etc. They gave us $2k to test app engine when we discussed how to use the platform
18:20:46  <kenansulayman>rescrv The issue is mainly that we run a high-concurrency realtime system and need to write up to 20 - 30 things per user per second (for instance a realtime editor)
18:20:59  <rescrv>brycebaril: not really. the algorithm doesn't really care about whether a write is an update/delete/first-insert. In fact, if you had such an algorithm that did care, you could turn everything into an append, and a delete would be an append that says a previous value is invalid.
18:22:04  <rescrv>kenansulayman: yeah, app engine doesn't offer that high of a perf, but the arch of spanner/big table can support many thousands of writes/s-tablet
18:22:37  <kenansulayman>rescrv Could you link to that? Google shows stuff that's most probably not "spanner"
18:23:15  <kenansulayman>rescrv ow it's from google
18:23:34  <rescrv>here's what I'd recommend watching/reading: https://www.usenix.org/conference/osdi12/elmo-building-globally-distributed-highly-available-database
18:24:37  <rescrv>there's plenty of spanner references floating around (back to '07 or earlier), but that's the definitive publication on how it works/what it does
18:24:49  <kenansulayman>rescrv it's neither open source nor a service, right?
18:25:34  <rescrv>kenansulayman: correct. I suspect it'll be closed for quite some time. I also suspect that any clones claiming to be open source will be nowhere near close to what Spanner really is.
18:26:37  <kenansulayman>That's pity. In order to get the stores right which enables our realtime collaboration, we should have been a storage architecture company on the first place ... :x
18:27:57  <rescrv>I'm not sure what exactly you're doing, but it sounds like you might benefit from doing in-memory replication, and persisting a log of what was done that can be replayed to restore state.
18:28:42  <rescrv>rather than trying to write the outputs of the computation (20-30 per user per second), write the inputs from the user, and make it deterministic.
18:28:52  <brycebaril>kenansulayman: or look into some of what dominictarr has done with https://github.com/dominictarr/scuttlebutt
18:30:25  <kenansulayman>rescrv That's why the in-memory concept of lmdb was a delight to us
18:31:23  <kenansulayman>brycebaril We already use that code for log rotation :) But master/slave for the rest
18:32:45  <rescrv>kenansulayman: I'm not talking storing a database in memory. I'm thinking more along the lines of a replicated state machine: http://www.cs.cornell.edu/fbs/publications/smsurvey.pdf
18:33:10  <rescrv>you have an in-memory application that's very fast. Not a stateless app writing to a db, but an actual application.
18:33:29  <rescrv>log the inputs to the state machine, and you'll always end up in the same state given the same inputs
18:33:48  <rescrv>so you're not bound by the speed of a data store, you're bound by the speed of whatever app you're writing
18:34:01  <rescrv>which I almost guarantee can be faster than storing/retrieving items from a data store
18:34:21  <rescrv>so there's an actual thread (and replicas of that thread) doing the work corresponding to the user
18:35:16  * thlorenzjoined
18:38:30  <kenansulayman>rescrv Are there existing solutions for that, so that we do not lose a high amount of HR on building that? :)
18:39:40  <rescrv>I don't know about existing solutions that let you maintain a cluster of state machines. I do know of at least one that lets you maintain several state machines in one cluster. You'd have to maintain several clusters independently, but it's easy scaling.
18:41:18  <kenansulayman>Which one?
18:41:32  <rescrv>https://github.com/rescrv/Replicant
18:41:37  <rescrv>Here's an example: https://github.com/rescrv/Replicant/blob/master/examples/log.c
18:41:43  <rescrv>that's the server-side code
18:42:11  <rescrv>here's the client header: https://github.com/rescrv/Replicant/blob/master/client/replicant.h
18:42:40  <rescrv>let's say you call send("myobj", "log", "abc", 4, ...)
18:43:29  <rescrv>it'll translate that into the call to log_log and it'll print "log was asked to log\"abc\" and it is 3 bytes long"
18:43:31  * thlorenzquit (Ping timeout: 245 seconds)
18:44:07  <kenansulayman>rescrvs Wouldn't that stateless log be a deadlock when the system goes down
18:44:14  <rescrv>underneath the hood, it takes care of maintaining the correct number of replicas of the "log" object, and will send that to it
18:44:23  <rescrv>what do you mean?
18:44:28  <kenansulayman>Because it has to be replayed?
18:45:39  <rescrv>there's two aspects to that. First, it maintains multiple replicas. You deploy 5, you can have two crash, and then another one crash later. So those replicas are kept live. It's only after you kill/restart everything that you have to replay the log
18:48:26  <rescrv>if it's that big of a deal, you can always take a snapshot that represents a prefix of the log, and replay from that point
18:51:26  <kenansulayman>That makes sense, yes.
18:52:45  <kenansulayman>But lets assume we have a hash-ring whereas each node is a memory-mapped database
18:53:18  <kenansulayman>I thing that makes more sense at this point, because on high-load writes the logs could grow infinitely, not?
18:53:42  <rescrv>not if you keep snapshotting/truncating it.
18:53:49  <rescrv>it's two differnet ways of building things.
18:53:56  <rescrv>you could put the state machines around a hash ring too
18:54:16  <rescrv>the state machine approach only makes sense if the app itself has a lot of amplification when persisted to a database
18:55:39  <rescrv>if the overhead of persisting to the database isn't that much, then the DB makes sense too
18:55:40  * DTrejojoined
18:57:36  <kenansulayman>rescrv We'll see where we go. We're not too much on users right now and will grow (around 1000 recurring users; we just started) That is, our peak is at max some thousand queries per second.. so I think we should focus on the general development for now.
18:57:59  <kenansulayman>rescrv However, I'd be extremely delighted to see HyperDex land in the "userspace" :)
18:58:59  <rescrv>I'm working hard to make that possible
18:59:03  <rescrv>I have to run for a bit though
18:59:07  <rescrv>it was good talking to you
19:00:13  <kenansulayman>rescrv Yes, thanks. I learned quite somethings and will read into the pdfs :)
19:01:22  <kenansulayman>rescrv btw https://developers.google.com/datastore/
19:04:39  * DTrejoquit (Remote host closed the connection)
19:05:06  * DTrejojoined
19:07:34  * Acconutjoined
19:09:28  * DTrejoquit (Ping timeout: 240 seconds)
19:12:25  * Acconutquit (Ping timeout: 272 seconds)
19:12:28  * wolfeida_joined
19:15:51  * wolfeidauquit (Ping timeout: 272 seconds)
19:20:23  * thlorenzjoined
19:25:05  * thlorenzquit (Ping timeout: 272 seconds)
19:37:40  * eugenewarequit (Ping timeout: 264 seconds)
19:39:49  * wolfeidaujoined
19:40:36  * rudquit (Quit: rud)
19:43:07  * wolfeida_quit (Ping timeout: 265 seconds)
20:20:50  * thlorenzjoined
20:21:35  * rudjoined
20:21:36  * rudquit (Changing host)
20:21:36  * rudjoined
20:25:02  * thlorenzquit (Ping timeout: 240 seconds)
20:40:46  * thlorenzjoined
20:45:28  * thlorenzquit (Ping timeout: 265 seconds)
21:21:12  * thlorenzjoined
21:25:26  * thlorenzquit (Ping timeout: 240 seconds)
21:38:10  <trevnorris>rvagg: what's going on?
21:49:18  * thlorenzjoined
21:53:03  * disordinaryjoined
21:53:26  * thlorenzquit (Ping timeout: 240 seconds)
22:04:35  * jcrugzzjoined
22:08:40  * thlorenzjoined
22:13:36  * DTrejojoined
22:15:22  * dguttmanjoined
22:16:28  * DTrejoquit (Remote host closed the connection)
22:17:02  * DTrejojoined
22:21:26  * DTrejoquit (Ping timeout: 245 seconds)
22:35:41  * DTrejojoined
22:40:25  * DTrejoquit (Ping timeout: 272 seconds)
22:47:51  * DTrejojoined
22:52:04  * DTrejoquit (Ping timeout: 246 seconds)
23:13:10  <levelbot>[npm] [email protected] <http://npm.im/nitrogen>: Nitrogen is a platform for building connected devices. Nitrogen provides the authentication, authorization, and real time message passing framework so that you can focus on your device and application. All with a consistent development platform that leverages the ubiquity of Javascript. (@tpark)
23:22:03  * thlorenz_joined
23:26:42  * thlorenz_quit (Ping timeout: 256 seconds)