| 00:07:29
| <rvagg> | chilts: I agree on incrementing ids, those need to die along with AUTO_INCREMENT when you move away from a traditional database |
| 00:09:30
| <rvagg> | I ended up in a few discussions last night that could be summarised as: "my application needs <feature> in a database database", "but isn't that because you've designed your application around your database?" |
| 00:09:39
| <rvagg> | unique, incrementing ids generally fit into that category |
| 00:09:44
| <rvagg> | BE GONE! |
| 00:19:36
| <chapel> | rvagg: I created a snowflake like id generator for using with level |
| 00:20:04
| <chapel> | though, its rough so I haven't published it |
| 00:20:33
| <chapel> | makes large keys though, 128bits |
| 00:20:59
| <chapel> | full timestamp, mac address and increment token when timestamp is the same |
| 00:21:10
| <rvagg> | cool, will be interesting chapel, see also https://github.com/chilts/flake |
| 00:22:03
| <chapel> | it supports up to 31,744 unique ids every millisecond |
| 00:24:00
| <chilts> | rvagg: absolutely, I think people need to move away from what they knew, to want they are going to know! :D |
| 00:24:06
| <chilts> | man, that's a quote if ever I saw it |
| 00:24:44
| <chilts> | chapel: yeah, see flake if you wanna compare, use or just copy :) |
| 00:24:55
| <chapel> | I saw flake, was part inspiration, but mixed with a little ObjectID (mongodb) |
| 00:25:02
| <chilts> | in fact, let me know if you have an idea to make flake better and I'll steal back! :D |
| 00:25:08
| <chilts> | cool |
| 00:25:10
| <chapel> | heh chilts |
| 00:25:30
| <chapel> | flake looks good, wanted to focus on binary/buffer usage in level |
| 00:25:51
| <chapel> | and wrap the ids so that you can grab info like the timestamp from it |
| 00:26:02
| <chilts> | I think I stole the idea from snowflake (Twitter) but changed it to not need a central server |
| 00:26:06
| <chilts> | ah binary, cool |
| 00:26:09
| <chapel> | chilts: yeah |
| 00:26:12
| <chilts> | wonder if I should make a binary version |
| 00:26:17
| <chapel> | chilts: you should :) |
| 00:26:26
| <chilts> | flake() and flakeBin() ... I guess it's just a conversion |
| 00:26:30
| <chilts> | without the '-' |
| 00:26:35
| <chilts> | or something |
| 00:26:42
| * chapel | needs to publish mine |
| 00:26:57
| <chapel> | then you can feel free to use anything if its worthwhile |
| 00:26:58
| <chilts> | or I'll pass that as an option to the generator ... flake('eth0', { format : 'bin' }) |
| 00:27:03
| <chilts> | cool, thanks |
| 00:27:30
| <chilts> | good idea ... hope you're happy for me to steal the idea of doing a bin version :D |
| 00:27:38
| <chapel> | no worries |
| 00:27:45
| <chapel> | I made mine for a project Im working on |
| 00:27:46
| <chilts> | so, I guess I should return a Buffer() |
| 00:27:56
| <chapel> | beyond that Id be happy for anyone to use the code however they see fit |
| 00:28:12
| <chilts> | I guess integers don't go that high (53 bits, rather than 128) ... ! |
| 00:29:24
| <chapel> | what do you mean? |
| 00:30:03
| <chilts> | sorry, I mean I'll have to return a Buffer() like we said, rather than a kind of native type (like an integer) |
| 00:30:14
| <chilts> | 'coz it's not high enough precision |
| 00:30:19
| <chapel> | yeah |
| 00:30:21
| <chilts> | col |
| 00:30:25
| <chilts> | +o |
| 00:30:35
| <chapel> | well, you have to use a buffer because thats the only way to compose the separate parts |
| 00:30:39
| <chapel> | :P |
| 00:30:51
| <chilts> | :) |
| 00:32:39
| <rvagg> | btw, being able to pass buffers directly into levelup -> leveldown -> leveldb ought to be the most efficient in terms of getting data down through the stack |
| 00:32:54
| <rvagg> | buffers don't require an additional memcpy that strings do |
| 00:33:09
| <rvagg> | so, buffers as keys, +1 if you can make it work and not be too awkward |
| 00:35:06
| <rvagg> | tho, buffers as keys will mess up many of the levelup extensions people have been building because they depend on keys being strings to do their magic, consider sublevel for instance, which serves as the basis for many extensions |
| 00:35:49
| <chapel> | well |
| 00:36:41
| <chapel> | the other tool that I found (thanks to mikeal) is https://npmjs.org/package/bytewise |
| 00:36:48
| <chapel> | which solves that issue rvagg |
| 00:37:08
| <chapel> | since you can throw strings in there, and the final key is a buffer, and sortable |
| 00:37:09
| <rvagg> | yeah, that's cool, that guy has a couple of other similar pkgs worth a look too |
| 00:37:32
| * st_luke | joined |
| 00:42:27
| <chapel> | rvagg: Ive been using bytewise and it works pretty well |
| 01:02:26
| * st_luke | quit (Read error: No route to host) |
| 01:04:25
| * st_luke | joined |
| 01:22:23
| * st_luke | quit (Remote host closed the connection) |
| 02:06:17
| <chapel> | chilts: did you make flake for leveldb, or what was the inspiration? |
| 02:08:39
| <rvagg> | chilts has dreams of building a Node dynamo |
| 02:10:26
| * st_luke | joined |
| 02:23:01
| * brianloveswords | quit (Excess Flood) |
| 02:24:57
| <chilts> | chapel: I think I first built it for fun one afternoon (took about 2 hours of playing) but with visions of using it in the future "for shits and giggles" |
| 02:25:05
| <chilts> | but it actually has some nice properties :) |
| 02:25:06
| <chapel> | ah, I do as well, though don't think I am knowledgeable enough atm to do the whole dynamo bit |
| 02:25:22
| * brianloveswords | joined |
| 02:25:24
| <chapel> | chilts: yeah, the structure of the ids are nice |
| 02:25:29
| <chilts> | yeah, it'll be like Dynamo but with some things missing and some new things as extra |
| 02:25:42
| <chapel> | btw, mine is structured <timestamp> <macid> <sequence> |
| 02:26:08
| <chilts> | nice ... I put the timestamp and sequence together since they were the things changing |
| 02:26:14
| <chapel> | since leveldb only supports one process at a time, felt the process id was unnecessary (in flask that is) |
| 02:26:59
| <chapel> | err |
| 02:27:03
| <chapel> | wtf, flask, flake |
| 02:27:04
| <chapel> | :P |
| 02:28:58
| * ramitos | quit (Quit: Textual IRC Client: www.textualapp.com) |
| 02:29:55
| <chilts> | you might have multiple processes with multiple LevelDBs on the same machine :) |
| 02:30:18
| <chilts> | but yeah, this is general, independent of LevelDB (but useful for it too) |
| 02:30:20
| <chapel> | sure, but they wouldn't be clustering together in a dynamo style |
| 02:31:01
| <chilts> | flake is independent of LevelDB, so that's why it needs to be there, flake garantees unique IDs - not just for LevelDB |
| 02:31:38
| <chapel> | I like flake, part of the reason I wrote my own was just to write something |
| 02:31:48
| <chilts> | no worries :) |
| 02:32:00
| <chilts> | just clarifying why it needed the PID too |
| 02:34:05
| * mrevil | joined |
| 02:35:14
| * st_luke | quit (Remote host closed the connection) |
| 02:44:57
| <chapel> | chilts: with the pid your ids would be 160bits I believe |
| 02:45:14
| <chapel> | guess thats a worthy tradeoff for the extra uniqueness |
| 02:48:58
| <chapel> | or, no, 144bits hmm |
| 03:21:39
| <chilts> | 128 I think |
| 03:23:26
| <chilts> | same as UUIDs |
| 03:37:30
| <chapel> | well, my id is 128 and it is the same sans the process id |
| 03:43:44
| <chapel> | I don't think its an issue tbh, just mentioning it |
| 04:02:24
| * mrevil | quit (Remote host closed the connection) |
| 04:32:55
| * mrevil | joined |
| 04:41:15
| * mrevil | quit (Ping timeout: 256 seconds) |
| 04:50:57
| <levelbot> | [npm] [email protected] <http://npm.im/levelweb>: A front end for leveldb (@hij1nx) |
| 05:19:50
| * timoxley | joined |
| 05:37:44
| * mrevil | joined |
| 05:44:03
| <hij1nx> | juliangruber: may have found an issue with doing deletes over multilevel |
| 05:44:08
| * mrevil | quit (Ping timeout: 255 seconds) |
| 05:47:30
| <levelbot> | [npm] [email protected] <http://npm.im/levelweb>: A front end for leveldb (@hij1nx) |
| 05:47:44
| <hij1nx> | juliangruber: possibly fixed ;) |
| 05:51:32
| <hij1nx> | juliangruber: ah, there does not appear to be test coverage for "del"... |
| 05:51:55
| <hij1nx> | ...which is the only thing seeming not to work for me. |
| 05:52:44
| <juliangruber> | hij1nx: :O shit :D |
| 05:52:50
| <juliangruber> | that should have just worked |
| 05:55:49
| <hij1nx> | juliangruber: ah, but i wrote a test, and guess what? i figured out what it was and multilevel works fine :) |
| 05:56:11
| <hij1nx> | juliangruber: oops, that came out wrong. |
| 05:56:44
| <hij1nx> | juliangruber: what i meant to say was, i wrote a test, and multilevel works fine. it was a problem with something else. |
| 05:56:57
| <juliangruber> | i see |
| 05:56:59
| <juliangruber> | cool :) |
| 05:58:00
| * Pwnna | quit (Read error: Operation timed out) |
| 06:40:42
| * mrevil | joined |
| 06:45:20
| * mrevil | quit (Ping timeout: 252 seconds) |
| 08:24:48
| * ChrisPartridge | quit (Ping timeout: 264 seconds) |
| 08:38:25
| * dominictarr_ | joined |
| 08:40:59
| * julianduque | quit (Quit: leaving) |
| 08:46:18
| * dominictarr | quit (*.net *.split) |
| 08:46:19
| * dominictarr_ | changed nick to dominictarr |
| 09:13:26
| * ralphtheninja | joined |
| 09:21:32
| * JakeV | joined |
| 09:21:46
| * Raynos | quit (Ping timeout: 264 seconds) |
| 09:23:43
| * dominictarr | quit (Quit: dominictarr) |
| 09:39:26
| * joachimhs | joined |
| 10:02:26
| * joachimhs | quit (Read error: Connection reset by peer) |
| 12:21:36
| * Guest28778 | changed nick to weetabeex |
| 13:53:21
| * werle | joined |
| 14:44:10
| * mrevil | joined |
| 15:04:22
| * ramitos | joined |
| 15:05:14
| * ramitos | quit (Remote host closed the connection) |
| 15:19:00
| * ramitos | joined |
| 15:26:43
| * webdesserts | joined |
| 15:47:03
| * Pwnna | joined |
| 15:53:25
| <werle> | juliangruber: https://github.com/jwerle/level-model |
| 15:59:59
| * owen1 | quit (Ping timeout: 260 seconds) |
| 16:13:10
| * owen1 | joined |
| 16:41:34
| <chapel> | werle: nice, I whipped up a quick model interface on a personal project, yours looks very clean (looking at the tests atm) |
| 16:42:16
| <werle> | chapel: thanks I'm aiming to have a very similar interface to Mongoose |
| 16:42:25
| <chapel> | werle: same here :P |
| 16:42:49
| <chapel> | werle: glad to see there are more like minded people |
| 16:43:01
| <werle> | chapel: we should collab ! |
| 16:43:28
| * mrevil | quit (Remote host closed the connection) |
| 16:44:01
| * mrevil | joined |
| 16:44:01
| <chapel> | yeah, that could be good |
| 16:45:15
| <werle> | chapel: is chapel your github username? |
| 16:45:38
| <chapel> | yeah, I don't have anything up there atm re: level + mongo interface |
| 16:46:17
| <werle> | cool just followed you :) |
| 16:48:28
| * mrevil | quit (Ping timeout: 245 seconds) |
| 16:48:35
| <chapel> | ditto |
| 16:53:55
| <chapel> | werle: found draft, that looks very good |
| 17:00:14
| <werle> | chapel: thanks dude! always looking for help to make it better if you're up for it |
| 17:11:06
| <chapel> | I think the major difference between level-model and what I have been doing is, I've been toying with an index like system |
| 17:11:31
| <chapel> | wonder if I can safely roll that out separately, that could be useful |
| 17:12:55
| <werle> | oh hmm |
| 17:13:00
| <werle> | I would like to see it |
| 17:13:08
| <werle> | it could be useful |
| 17:16:23
| * ramitos | quit (Remote host closed the connection) |
| 17:16:33
| <chapel> | the only problem I see is reliance on how the keys are written |
| 17:16:56
| <chapel> | I guess it shouldn't matter, just keep the index part simple, where it creates a key and value set based on inputs |
| 17:17:12
| <chapel> | how the key is written isn't a concern of it |
| 17:22:34
| <werle> | how should the keys be written? |
| 17:37:14
| <chapel> | well, as a buffer for one |
| 17:37:24
| <chapel> | and also I am using bytewise |
| 17:38:03
| <chapel> | https://npmjs.org/package/bytewise << which makes it easy to compose binary keys that are sortable |
| 17:45:20
| <werle> | oh shit |
| 17:45:21
| <werle> | very nice |
| 17:45:37
| <chapel> | werle: in my prototype, single indexes work great |
| 17:46:04
| <werle> | chapel: looking forward to checking it out, please keep me posted. |
| 17:46:12
| <chapel> | will do |
| 17:46:26
| * chapel | is working on an objectid like binary id |
| 17:46:41
| <werle> | hmmm that may be very useful |
| 17:46:45
| <werle> | hmm |
| 17:47:00
| <werle> | maybe you can write a module that does just that :) |
| 17:47:12
| <chapel> | well I already have it as a module, just haven't published it |
| 17:47:20
| <chapel> | was tuning up the speed a bit |
| 17:47:44
| <chapel> | have it at ~700k id's a second |
| 17:48:40
| <werle> | very nice |
| 17:48:45
| <werle> | I can't wait ! |
| 17:51:19
| * JakeV | changed nick to Raynos |
| 17:56:26
| <chapel> | anyone here know why removing a function definition that isn't being used for anything would reduce the amount of operations done in a second? |
| 18:02:32
| <werle> | hm |
| 18:02:34
| <werle> | that is strange |
| 18:04:31
| <chapel> | its probably a peculiarity with v8 |
| 18:05:59
| * webdesserts | quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client) |
| 18:12:00
| <werle> | yeah something like that. doing something behind the scenes with the delete |
| 18:26:06
| * ramitos | joined |
| 19:53:52
| <werle> | https://github.com/jwerle/level-model |
| 19:53:55
| <werle> | more progress |
| 19:53:59
| <werle> | works in browser and node |
| 19:54:15
| <werle> | just needs a Levelup api |
| 19:54:29
| <werle> | tested with level-js |
| 19:55:09
| <werle> | I had to patch leveljs though because of turns empty arrays into empty strings |
| 19:55:09
| <werle> | https://github.com/jwerle/level-model/blob/master/leveljs.js |
| 20:14:03
| * dominictarr | joined |
| 20:24:15
| * dominictarr | quit (Quit: dominictarr) |
| 20:27:48
| * dominictarr | joined |
| 20:37:14
| * ralphtheninja | quit (Ping timeout: 272 seconds) |
| 20:38:41
| * ralphtheninja | joined |
| 20:49:29
| * dominictarr | quit (Quit: dominictarr) |
| 20:53:24
| * mrevil | joined |
| 21:17:30
| * werle | quit (Quit: Leaving.) |
| 21:21:40
| * ramitos | quit (Quit: Textual IRC Client: www.textualapp.com) |
| 23:01:54
| <Raynos> | idea! |
| 23:02:01
| <Raynos> | Store, gather, index & search gists in leveldb |
| 23:02:21
| <Raynos> | I want a quick way to search my gist for `function either() { ... }` because I know i wrote something like that and dumped it there somewhere |
| 23:03:36
| <Raynos> | then I want to expand that to code search through npm |