ToroDB internals: open source Java database middleware under the hood

ToroDB Stampede is an open source middleware that speaks the MongoDB protocol and replicates data live from MongoDB to an RDBMS. Data is transformed on-the-fly from a document model into a pure relational model, not a single flattened table. Schema is created automatically, there’s no need to design any tables beforehand. ToroDB is built 100% with Java8, and this talk is about its internals, how some significant problems are solved and what techniques and frameworks are used and why, such as:

• Network I/O with Netty, off-heap buffers and memory consumption.
• How D2R (Document-to-Relational) transformation is performed.
• How DDL changes are performed, isolation challenges and why some more naive considered alternatives do not work.
• How CUD (Create, Update, Delete) changes are transformed and applied into batches, as they are consumed from the MongoDB replication stream. Use of Akka Streams.
• How ToroDB Stampede deals with sharding.
• Docker integration into maven for fully automated unit and integration testing scenarios, including clusters of servers, failover, etc. JUnit 5 taken to the extreme!

Come and join this session to see how a high-performance database system can be built with Java. And of course contribute your own ideas and expertise, we’re open source and welcoming PRs!