SAPO Codebits 2009

Session Title

The BlackRay Data Engine

Abstract

The BlackRay Data Engine is an open source, high-performance, in-memory relational database, designed for large data sets and constant performance. Originally designed for directory applications it offers features such as token search, token position, phonetic search and the combination of these features with leading, trailing and mid-span wildcards. It is built to run on standard hardware, but offer the ability to index data in the over 100 Million row range, with constant search throughput of several 100 queries per second, even with complex queries.

In this talk we would like to explain our motivations for designing and building BlackRay, and then elaborate on the architecture of the internals of the data engine. The index structures inside the data engine are designed for low memory consumption, and the ability to quickly index large amounts of data. A total of five index layers, also called index perspectives, is required to fulfill the functions required for our search algorithms. Our smart combination of binary- and permuterm based search offers significant performance benefits over many traditional tree- and trie-based searches. Finally, searching for a token combination within a single table column only result in linear complexity, rather than exponential as in most typical index structures. The option to additionally compress the index further reduce the amount of memory used during operation.

Language

English

Speakers

Felix Schupp

Felix has been at the helm of SoftMethod since 2004. He received his degree in Computer Engineering from San Jose State University in 2000. Subsequently he was co-founder and Chief Operating Officer of PhysicianPal Inc, a San Francisco based solution provider for the healthcare industry, connecting insurances, workers compensation agencies and ambulatory surgery centers. From 2000 until 2003 he served as an Instructor and Program Manager for UC Berkeley International programs, focusing on advanced Internet business engineering. He has been an active participant in the open source community, and has been a contributor at several industry association congresses, most recently the EDM08 in London. Currently he is involved in getting the E.115v2 protocol standardization process with the ITU wrapped up. He is also a sponsor and mentor in the academic cooperative project "Telling Test Stories" with the University of Innsbruck. In his rare spare time felix enjoys getting some wind into the sails of his 1962 dragon "Sonnenschein". (more)