Going for a Lua(jit) rewrite

After 2 years of rewriting Rubyk and the Oscit library, things are still in “alpha”. This “failure” happens for two three reasons.

Trouble sources

Sharing before building

The first is the energy drawn by trying to create a very important part of Rubyk (network layer) as a collaborative effort within an existing community (open sound control). I kept worrying about latency and guaranteed delivery of packets and spent a lot of time discussing array formats and the existence or not of a Hash type.

Language design in C++

The second part that took a hell of a lot of time is the design of the dynamic typing, introspection capabilities and garbage collection to cope with the dynamic nature of the signals and objects in Rubyk. Doing all this work ended in a true language design and I do not have the competences nor the commitment (or wish) to do create such a language out of the void.

Multi-threading

The third point is a bad understanding of concurrent programming. When I started the whole thing over two years ago, I thought “let’s be ready for multi cores, we need threads”. So I carefully rewrote everything to be thread-safe. This new level of complexity is also hindering progress and I feel it is inherently bad (more on this later).

Getting out of trouble

Doing all these things wrong was very formative and I feel I am much better armed now to make the correct decisions (and the necessary tools are available now).

Use ØMQ

The first decision was to use zeroMQ for inter-process and network communications. It’s fast enough, packet delivery is guaranteed and I can send whatever I want.

Use Lua(jit)

The second decision was to rewrite the core in Lua (or LuaJit where possible) instead of C++. This means that we have a share nothing architecture between threads and that the development will be much easier and agile. We can also delegate all the trouble of dynamic typing, introspection and memory management to Lua.

Lua will be much more fun to work with, can easily be interfaced with any C code and is really fast enough for the “glue” between objects that the Rubyk core provides.

In order to decide between a closures based implementation (uses more memory, elegant) and an implementation based on metatables (method resolution resembling Ruby classes), I did some tests with different implementations of the Outlet/Inlet. The tests are pretty trivial, so please do not consider the results below as anything else then a starting point on the subject.

The memory test creates 100'000 mixer objets (they add their inputs and send to output).

The speed test executes an input+send in a connected graph 10'000'000 times.

implementation	Lua memory	LuaJit memory	Lua speed	LuaJit speed
closures (no self)	95.6 Mb	62.6 Mb	11’821 ms	355 ms
metatable	40.4 Mb	29.8 Mb	17’004 ms	323 ms

Lua test files: closures.lua, metatables.lua.

The first tests where flawed since the compiler optimized the code to an empty loop. Thanks to Mike Pall (author of LuaJit) for pointing this out.

From the tests (and the remark by Mike), the “metatable” route is the way to go since it uses less memory and enables more optimizations from the LuaJit compiler.

Conclusion

I am confident that these choices will remove what I don’t like in working with Rubyk and let me focus on the pieces of the software that really matter: reusable components, graph design, remote user interface, network transparency, embedding, etc.

Thanks to Lua(jit) and ØMQ for the great projects !

PS: I just need to find or write a simple cross-platform GUI that is easy to build and script (Qt Quick maybe).