I've been fairly silent in public around Emdros for a couple of months.  That is not because Emdros has not required my attention.  On the contrary, I've been feeding it lots of carrots and hay so that its engine "horsepower" can grow.

In particular, this is a preliminary announcement that I have written an entirely new database backend for Emdros -- to parallel SQlite 2/3, MySQL, and PostgreSQL.  I call the new database backend the "Bit Packed Table" backend, or BPT for short.

The upshot of the new database backend is that it beats the pants off all other currently implemented backends in terms of query speed.  I will post some query statistics results later.

In addition, the database size shrinks by quite a margin compared to even SQLite 3, which compresses data already. In my tests, the database size can shrink between 42% and 61% as compared to the same data in SQLite 3.

So how did I do this?

First of all, the BPT backend is tailored specifically to the needs of the EMdF model. It doesn't even attempt to be a full SQL engine with a general relational model. Instead, it focuses on the requirements and inherent properties of the EMdF model.

Secondly, the BPT engine is simple. Because it does only what is necessary for the requirements of the EMdF model (and uses some fairly clever storage schemes tailored to the EMdF model), it is quite simple both to maintain and to understand.  It is coded in about 9000 lines of well-commented C++.

Thirdly, it uses fairly old techniques (bit packing obviously being one of them) to store the data.

Fourthly, the BPT backend uses fairly recent results from research by Allison L. Holloway and David J. DeWitt, "Read-optimized databases, in-depth" (Proceedings of VLDB 2008) that show that read-optimized, row-oriented databases can be quite fast, even when compared to column-oriented databases, which otherwise are (partly) all the rage these days.

The BPT engine will stay proprietary for now, and won't be Open Sourced for the foreseeable future. Existing and prospective licensees are welcome to drop me a line at http://emdros.org/contact.html

Ulrik