Record/playback data streams

I have just finished creating a stream recorder/playback engine.

For several months, I have been investigating various storage solutions ranging from SDIF to Matroska. These are really great tools with lots of optimization to improve seek and storage. These enhancements come at a price: library usage complexity.

Since I do not have enough time right now to wrap my head around these huge and not easy to use libraries (Matroska is not a library: I am talking about the libraries implementing ebml) I decided to work with what I really know well: SQL databases. In this case, since I want to store the data in files with a single client the choice is evident: SQLite.

So it took me half an hour to write the tests so I could see clearly what kind of interface I wanted to use, an hour to to integrate Lua bindings (LuaSQLite) and compile everything with sqlite3. Once all this was in place (“sqlite3” is now a module in Lubyk), I could just write my recorder/player in Lua and things went on really easily, especially because I could use “worker:now()” to get the current time and “rk.Timer” to handle timing issues during playback.

I have adapted the “wii_cube” example to record wiimote movements and it’s very smooth (taking less the 2% CPU with OpenGL and all).

As a teaser, here’s an example with the db.Stream recorder/playback engine:

set (create/update)

streamer = db.Stream() -- in memory db

streamer:set({t=10, x=4, y=8})
streamer:set({t=10, z=9})
-- these two lines create a single event at time '10' with
-- three tracks "x", "y" and "z" 
streamer:at(10) ---> {t=10, x=4, y=8, z=9}

record

Record is like “set” but it uses the current time from worker to set “t”. You must first call “rec_start()” or the “rec” method is a noop (this is needed so that the “rec” method can be dropped in a signal flow and only record when needed.

streamer = db.Stream() -- in memory db
streamer:rec_start()

streamer:rec{x=4, y=8, z=9}
...
streamer:rec{x=5, y=12, z=7}
...
streamer:rec{btn='hop'} -- set strings is ok

read

Get next timestamp (useful to advance one event at a time):

-- if t holds the current playback position, calling next will
return the next timestamp or nil
next_t = streamer:next(previous_t)

Get event content:

row = streamer:at(t)

print(row.t, row.x, row.y, row.z, row.btn)

Get track data in range [a, b[:

track = streamer:track('x')

for row in track:range(a, b) do
  -- draw point in OpenGL for dataset "x" 
end

The last method (range) will be used to plot signal with OpenGL and avoids loading all data in memory through the use of an iterator.