07:31:48  * rendarjoined
15:58:21  <creationix>SinisterRectus: neat, hadn't seen yours
15:59:13  <creationix>SinisterRectus: do you know how I could use the xchacha20 encryption on a coro-net stream? I'm still trying to figure this out
15:59:54  <creationix>the protocol I'm implementing sends the first message in plain-text since it contains the crypto nonce, but then everything after that is encrypted.
16:00:00  <creationix>The key is obtained through a side channel
16:33:58  <SinisterRectus>creationix: i use it for decoding and encrypting/decrypting audio data for discord, but i just read a raw udp socket. i haven't tried using coro-* yet, but i plan to try in the near future
16:34:10  <SinisterRectus>well, decoding is something different
16:35:18  <SinisterRectus>write to udp* rather
16:35:21  <SinisterRectus>i don't read yet
16:37:03  <SinisterRectus>one thing that kind of bums me out is (l)uv only accepts lua strings, so if you want to move data between an ffi function and a (l)uv function, you have to eventually ffi.string it
16:39:35  <SinisterRectus>i assume there is no way to skip that step without also using an ffi binding for uv
16:40:47  <SinisterRectus>unless you can pass cdata to C from lua
17:00:11  <creationix>I can modify luv to accept ffi cdata
17:00:19  <creationix>I thought I had this already, but I guess not
17:00:27  <creationix>the problem is knowing the length
17:00:45  <creationix>a plain "uint8[?]" from ffi doesn't embed it's length as far as I know
17:01:19  <creationix>I think it supports luvit buffers, let me see...
17:02:39  <creationix>hmm, maybe not https://github.com/luvit/luv/blob/master/src/stream.c#L168-L178
17:03:29  <creationix>yep, string only https://github.com/luvit/luv/blob/master/src/stream.c#L21
17:04:33  <creationix>it was my duktap bindings that supported cdata and strings https://github.com/creationix/seaduk/blob/master/src/duv/stream.c#L62-L76
17:04:37  <creationix>*duktape
17:05:26  <creationix>that works because duktape has a native "buffer" type with length embedded https://github.com/creationix/seaduk/blob/1c740975fdd932ec1c92e48c58dcdd004ec355f2/src/duv/utils.c#L272-L279
17:07:37  <creationix>looks like ffi.sizeof(ctype) supports getting the length. Maybe I can use that from luv somehow
17:34:31  <SinisterRectus>hmm
17:35:45  <SinisterRectus>oh that's neat
17:35:54  <SinisterRectus>> ffi = require 'ffi'
17:35:54  <SinisterRectus>> arr = ffi.new("uint8_t[?]", 4)
17:35:54  <SinisterRectus>> ffi.sizeof(arr)
17:35:54  <SinisterRectus>4
17:36:33  * SkyRocknRolljoined
17:36:51  <creationix>so I think it can be done. Just a question of how to access that from C
17:36:59  <creationix>I don't want to have to eval a lua string just to get the length
17:37:20  <creationix>I could cache a reference to ffi.sizeof the function and call it I guess
17:37:34  <creationix>still, I wonder if luajit has an internal C function I can call directly
17:38:54  <creationix>looks like a private function https://github.com/LuaJIT/LuaJIT/blob/master/src/lib_ffi.c#L591-L611
17:40:12  <creationix>it's `_lj_cf_ffi_sizeof` in the luvit binary
17:41:07  <SinisterRectus>so there's no c api for the ffi
17:41:21  <creationix>not a supported one at least
17:41:30  <creationix>there is nothing technically stopping us from calling the internal functions
17:41:38  <creationix>it just might change at any time
17:41:40  <SinisterRectus>true
17:41:58  <creationix>luajit development is pretty slow and for luvi at least, we statically include luajit
17:42:24  <creationix>but for luv published to luarocks, luajit 2.0.x vs 2.1.x are both used a lot
17:42:35  <creationix>and luaffi on PUC lua is probably different too
17:43:12  <SinisterRectus>well. all luv functions that accept strings also accept a table of strings, right? or is that only a few of them
17:43:32  <creationix>not all, just ones where libuv accepts an array of buffers
17:43:42  <creationix>(using the writev primitives at the syscall level)
17:43:59  <SinisterRectus>ah. i was thinking what if they also accepted {cdata, len, cdata, len ...}
17:44:18  <SinisterRectus>kind of hard to deal with that though
17:44:23  <SinisterRectus>requires a few assumptions
17:44:32  <creationix>we could do that, just define a convention and document it
17:44:51  <creationix>I really like passing in the cdata directly now that I know it does store the length somewhere
17:44:57  <creationix>just a question of how to best support that
17:45:08  <SinisterRectus>i assume the overheard of creating a {cdata, len} tuple would be less than making a lua string especially for large strings
17:45:27  <creationix>perhaps, it's hard to tell
17:45:38  <creationix>you're probably right for large strings. The interning needs to hash the string
17:46:00  <SinisterRectus>i agree, passing cdata alone would be great
17:47:01  <SinisterRectus>raises the question about returning cdata though :/
17:47:21  <SinisterRectus>would be nice if one could avoid lua strings entirely for streams
17:47:52  <SinisterRectus>i think that complexity is why i never brought this up before
17:48:15  <SinisterRectus>you'd basically need two luv apis in that case
17:48:28  <creationix>yeah, it would break enough things that assume string APIs
17:48:49  <creationix>maybe adding cdata versions of certain libuv APIs would be a good way to do it
17:49:14  <creationix>but how do you create cdata from C?
17:49:27  <SinisterRectus>i know a binding exists https://github.com/pguillory/luajit-libuv i just never looked into it because luv is too convenient
17:49:44  <creationix>well, that's different
17:49:57  <creationix>luajit ffi is *really* slow with callbacks as far as I know
17:50:06  <creationix>and when I started this, the ffi didn't support callbacks at all
17:50:10  <SinisterRectus>ah
17:50:27  <creationix>but since then, luajit added callback support to it's ffi and libuv added some convenience functions needed to write pure ffi bindings
17:50:47  <creationix>it might be fun to prototype and see how it actually performs
17:51:00  <SinisterRectus>latest commit 3 years ago
17:51:16  <creationix>There is also this point that daurnimator brought up https://github.com/daurnimator/luasodium/issues/1#issuecomment-320497706
17:51:51  <SinisterRectus>oh i agree entirely
17:53:32  <creationix>technically luajit ffi is amazing, but it does have it's downsides for sure
17:53:38  <SinisterRectus>if i had the time, i would write pure bindings for all the C that i used, but it was just too easy to write ffi bindings for the subset of libopus and libsodium that i needed
17:54:12  <creationix>in some ways ffi bindings are more portable. They are just lua
17:54:25  <creationix>I don't have to distribute binaries for each platform or have people compile C code on their machine
17:54:45  <SinisterRectus>true, but you still need the C libraries
17:54:46  <creationix>but especially when you get down to low-level system stuff, ffi authors just guess magic numbers
17:55:03  <creationix>the goal with luvi is to include all C libraries needed at that layer
17:55:22  <creationix>I wanted a build system where you could check the C libraries and versions you wanted and it would generate a custom luvi for you
17:55:33  <SinisterRectus>i don't expect luvi will ever include libopus and libsodium then
17:55:39  <creationix>so your package.lua could declare your C deps and your lua deps.
17:55:59  <creationix>libsodium maybe, especially if I end up using luvit at work
17:56:06  <SinisterRectus>how does luarocks handle C deps, i've never actually looked into it
17:56:12  <creationix>we're node and go at the moment, but luvit could work really well
17:56:26  <creationix>luarocks has several make-like backends it supports
17:56:34  <creationix>luv, for example compiles using cmake on luarocks
17:56:50  <SinisterRectus>but lit only supports lua
17:56:51  <creationix>the user has to have a c compiler and cmake installed before they can install luv via luarocks
17:57:15  <creationix>yeah, lit doesn't try to solve the compile-on-demand use case. It does have support for pre-built binaries though
17:57:29  <SinisterRectus>sure
17:57:34  <creationix>for example https://github.com/creationix/lit-tweetnacl
17:58:03  <SinisterRectus> main = "$OS-$ARCH/luatweetnacl.so",
17:58:05  <creationix>that one embeds a C bindings to tweetnacl
17:58:15  <SinisterRectus>why is it main
17:58:23  <creationix>https://github.com/creationix/lit-tweetnacl/blob/master/package.lua#L13
17:58:42  <creationix>it includes all arches when publishing the package, but when installing it as a dependency, it only includes the native version
17:58:59  <SinisterRectus>interesting
17:59:01  <creationix>the path convention is up to the author
17:59:10  <creationix>lit just provides the variables in the files section
17:59:33  <creationix>they match the ffi.os and ffi.arch variables in luajit
18:00:11  <creationix>and luvit's require will extract the libraries to a temp folder if they are embedded in the zip so it can dlopen them
18:00:52  <SinisterRectus>can it resolve a windows .dll vs a linux .so
18:00:53  <creationix>here is one using ffi bindings https://github.com/creationix/lit-blake2b/blob/master/init.lua
18:01:10  <SinisterRectus>oh haha
18:01:25  <creationix>the `module.action` helper extracts any arbitrary resource from the zip to a real file on the temp folder so you can do something with it
18:01:29  <creationix>https://github.com/creationix/lit-blake2b/blob/master/init.lua#L12
18:01:53  <creationix>the path resolving is up to the module author, lit merely provides the variables
18:02:09  <creationix>and luvit's require system provides tools to extract to temp folders
18:02:42  <SinisterRectus>how is module:load and module:action defined... is that in the metatable for module
18:02:53  <creationix>the only issue I've hit with this is it's not fine-grained enough. armv6 and armv7 are the same to ffi.arch
18:03:08  <creationix>ffi.os is just "Linux" for both glibc and musl
18:03:32  <creationix>that's a custom global from luvit's require, not lua's global `module`
18:03:43  <SinisterRectus>yes of course
18:03:54  <SinisterRectus>i just learned that it has a metatable though
18:03:59  <SinisterRectus>i knew the table existed
18:04:11  <creationix>yeah, there are some interesting helpers in there
18:05:00  <SinisterRectus>so it could be possible to unload a module
18:05:17  <creationix>maybe, not sure
18:05:28  <creationix>it's not a use case I tried to support
18:05:39  <creationix>we had a lot of issues when npm and node tried to support that
18:06:26  <SinisterRectus>require.cache is exposed in node though
18:07:12  <creationix>yep, and they regretted it ever since
18:07:15  <SinisterRectus>ah
18:08:21  <SinisterRectus>because i was messing with hot (un)loading code and i ran into not being able to unload a required module so i either had to use luvit-loader and package.loaded or manually loadstring the code
18:08:47  <creationix>yep
18:10:55  <SinisterRectus>anyway, re: cdata, it's not super important to me that cdata args be supported if cdata cannot be returned in the same api
18:11:28  <creationix>yeah, I think new cdata APIs on top might be the easiest path there
18:12:15  <creationix>`stream:writeC(cdata, [callback])` or something
18:12:21  <SinisterRectus>aye
18:12:39  <creationix>`stream:readStartC(callback)`...
18:12:49  <SinisterRectus>likewise
18:13:01  <creationix>doing just a couple functions in stream will get you 95% for network stuff
18:13:09  <creationix>read and write
18:13:38  <creationix>dgram might need it's own since it's not a stream
18:13:45  <creationix>then the fs stuff, macros could help there
18:14:17  <creationix>the other option is add an option to read function and make write function accept cdata
18:14:31  <creationix>or set a per-object option
18:18:30  <SinisterRectus>all of which would be useless for non-luajit lua (or lua without a copy of the ffi)
18:18:40  <creationix>yep, it would need ifdef
18:19:26  <creationix>maybe just a convention of {cdata, length} instead of string
18:19:41  <creationix>could even allow {string, length, offset} for slicing of strings
18:19:53  <creationix>just need to make it unambigious from vector of values
18:20:20  <creationix>different options
18:20:30  <creationix>at a minimum, I'd like luvit's pretty printer to show cdata better
18:20:43  <SinisterRectus>now my mind is wandering over all the options. what about a built-in buffer userdatum
18:20:46  <creationix>maybe if it's `uint8_t` show it like node buffers
18:21:01  <creationix>yeah, luvit has a built-in buffer
18:21:09  <creationix>it's just a table with ffi cdata and explicit length in lua
18:21:16  <SinisterRectus>i mean built into luv as lua userdata
18:21:23  <creationix>and a metatable with node Buffer-like methods
18:21:36  <creationix>yeah, we could do that
18:21:54  <creationix>have a utility for converting ffi cdata to it
18:21:57  <SinisterRectus>type() would be 'userdata' instead of 'cdata' so it would be portable across all luas
18:22:06  <creationix>then luv functions wouldn't have to support cdata
18:22:38  <SinisterRectus>tostring() could return a lua string
18:22:50  <SinisterRectus>but you could pass the userdata around and manipulate it as userdata
18:22:59  <creationix>yep
18:23:08  <creationix>and we could even make it resizable if we wanted
18:23:22  <creationix>just needs a layer of indirection in the C struct so realloc works
18:23:44  <creationix>sure such libraries already exist for lua
18:24:26  <SinisterRectus>yeah. i'm liking the buffer userdata idea best.
18:25:03  <creationix>supposedly auxlib has buffers, trying to find it
18:26:00  <creationix>See the `luaL_Buffer` type here https://www.lua.org/manual/5.2/manual.html
18:26:32  <SinisterRectus>oh. hmm.
18:27:05  <creationix>yep, 5.1 has it too https://www.lua.org/manual/5.1/manual.html#luaL_Buffer
18:27:39  * rendarquit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
18:28:13  <creationix>it's not quite what I want, but may be useful
18:28:26  <creationix>seems an optimization for building large strings our of many small strings
18:30:21  <SinisterRectus>hmm. the original issue is that ffi bindings like my libsodium binding returns cdata and to use this with luv, i have to make a lua string before passing it.
18:30:26  <creationix>if we made our own and didn't want it resizable, a simple VLA struct as userdata would work great
18:30:32  <creationix>https://www.irccloud.com/pastebin/JEnXfjE8/
18:30:42  <SinisterRectus>would a buffer userdata solve this, or is that just moving the issue somewhere else
18:31:04  <creationix>right, you would have to convert your cdata to one of these instead of converting to a string
18:31:13  <creationix>should be faster, especially if we can avoid a copy
18:31:36  <SinisterRectus>uv.new_buffer(cdata)
18:31:46  <SinisterRectus>uv.new_buffer(cdata, len)
18:31:56  <creationix>yep, something like that
18:32:12  <SinisterRectus>you'd have to account for different types, though
18:32:16  <creationix>do you think resizable is desired? I guess it could be added later
18:32:44  <SinisterRectus>a buffer only makes sense if the cdata is an array or pointer i guess
18:32:57  <SinisterRectus>i've never resized a buffer in my brief dealings with them
18:33:07  <creationix>https://www.irccloud.com/pastebin/dwWCrXOZ/
18:33:39  <creationix>though having the indirection also allows pointing to existing c memory
18:33:49  <creationix>we just need to keep the GC for the cdata alive
18:34:15  <creationix>copying to a fixed luv buffer is easiest
18:34:18  <creationix>and still cheaper than interning as a lua string
18:34:31  <creationix>but copy is always expensive for high-traffic network stuff
18:37:40  <SinisterRectus>as long as strings can be avoided, i'm happy. it hurts to have to use lua strings for udp reading/writing encoding/decoding encrypting/decrypting an audio stream
18:38:40  <SinisterRectus>i have to head out now though. let me know if you have any amazing revelations
18:38:56  <creationix>ok
18:40:59  * SkyRocknRollquit (Remote host closed the connection)