Sharding destroys the goals of your relational database

Sharding does destroy your relational database – which is a good thing. The idea behind sharding is to distribute data to several databases based on certain criterias. This could for example be the primary key. All entities that keys begin with 1 go to one database, with 2 to another and so on (often modulo functions on the key are used, or groups based on business data like customer location, or function). Several reasons exists for sharding, the main two being better performance and lower impact of crashed databases – only persons with a name that starts with S will be affected by a database crash.

Relational databases were the tool of choice for several decades when it comes to data storage. But they do more than store data. Even reading operations can be split into several functions. There are at least three kinds of database read queries:

  • Data graph building queries: With these you get your data out of the database, customers together with adresses etc.
  • Aggregation queries: How many orders have been stored in the August, aggregated by product category
  • Search queries: Give me all customers who live in New York

Sharding now does away with the second and third query and reduces databases to data storage. Because the shards are different databases on different systems you can’t aggregate queries (compared to a cluster) without custom code across systems and you cannot search with one query (only several ones – one to each database). Databases have lead to the notion that search and retrieval are linked together and should be dealt together. Most people think as retrieval and search as the same thing. This has blocked development on technologies. Sharding, S3, Dynamo, Memcached have changed this preception recently. I’ve written about splitting search and retrieval in “The unholy legacy of databases”. There I quote Rickard from Qi4j fame:

Entities are really cool. We have decided to split the storage from the indexing/querying, sort of like how the internet works with websites vs Google, which makes it possible to implement really simple storages. Not having to deal with queries makes things a whole lot easier.

and have concluded

Free your mind! Storage and search are two different things, if you split them, you gain flexibility.

People talked about splitting storage and search for some time now. Search engines like Lucene have driven searching out of databases. But mainly the notion of store&search is prevalent. Sharding as a mechanism for more perfomance and lower risk will move into many web companies and reduce databases to storage mechanism and drop the aggreation (data warehouse and reporting) and search parts. Those can be better filled with real data warehouse servers like Mondrian and search services based on Lucene or semantic enginse like Sesame. And storage might move from databases to simple storages like Amazon Elastic Block Storage or JDBM.

Thanks for listening, and think about your databases.

The unholy legacy of databases

When reading about the status of Qi4j on Rickards blog, I stumbled about

Entities are really cool. We have decided to split the storage from the indexing/querying, sort of like how the internet works with websites vs Google, which makes it possible to implement really simple storages. Not having to deal with queries makes things a whole lot easier.

We made the same experience when we developed the SnipSnap wiki application several years ago. We had a split in storages and search, each part with it’s own Java interface (a component could implement both of course). This way we could have Lucene, database and in-memory search and database and file (XML, plain text) storage. We were very flexible with storage and search this way and people could easily implement different storage backends because developers have been freed from the search implementation. Rickard seems to have made the same experiences:

We have one EntityStore based on JDBM (persistent binary hashmap), one on JGroups (replicated cluster hashmap), one on Amazon S3 (for global storage), and one on iBatis (for RDBMS storage)

So today SnipSnap would easily be able to supply a S3 backend, because of the split, whereas others which rely on the storage/search combination have much more problems to support a storage-only backend. So they have problems to support S3 or WebDav out of the box.

Why don’t more people split the problem of storage into storage and search? After some contemplation on the topic, perhaps it’s the unholy legacy of databases. Databases make it easy to solve the search/storage problem with only one technology. After 30 years of databases the problems have merged in a way that most developers think of them as one problem. By splitting the problem again, projects will be freed for better backends and better search solutions. Open Source projects will emerge which adress each of the problems better than current databases do.

This of course breaks the DAO pattern and the usage of the EntityManager as an DAO replacement and should be replaced by a Storage and Search pattern. Free your mind! Storage and search are two different things, if you split them, you gain flexibility.

Thanks for listening.