Comments on: Essential storage tradeoff: Simple Reads vs. Simple Writes

By: Games Outside the Box, Analog Computers, Read-Write-Costs etc. | Gamlor

Games Outside the Box, Analog Computers, Read-Write-Costs etc. | Gamlor — Thu, 11 Mar 2010 23:38:06 +0000

[...] an old post about the read-/write-cost tradeoff. For most applications reads occur way more often than writes. Therefore this application should [...]

By: SQL is Dead. Long Live SQL. : Dataspora Blog

SQL is Dead. Long Live SQL. : Dataspora Blog — Thu, 26 Nov 2009 02:10:46 +0000

[...] The second is a rejection of the strong typing of relational schemas, which make changes to data models, which are inevitable, disastrously difficult to achieve. It also makes writing to the data store a complex process. [...]

By: Scalable Database Links « streamhacker.com

Scalable Database Links « streamhacker.com — Mon, 26 Oct 2009 16:20:26 +0000

[...] Code Monkeyism: Essential storage tradeoff: Simple Reads vs. Simple Writes [...]

By: Ennuyer.net » Blog Archive » Rails Reading - Sept 21, 2009

Ennuyer.net » Blog Archive » Rails Reading - Sept 21, 2009 — Mon, 21 Sep 2009 15:16:56 +0000

[...] Code Monkeyism: Essential storage tradeoff: Simple Reads vs. Simple Writes [...]

By: Curt Adams

Curt Adams — Sat, 19 Sep 2009 08:17:43 +0000

If the denormalized views are constructed from a normalized database, the normalized data isn’t dropped. It’s still there, and can be used to verify or reconstruct documents. My understanding of the NoSQL movement is that it abandons the goal of a normalized data structure entirely.

By: Jason Warner

Jason Warner — Fri, 18 Sep 2009 19:18:22 +0000

If people haven’t, now would be a great time to go and review Brewer’s CAP Theorem!

http://www.julianbrowne.com/article/viewer/brewers-cap-theorem

By: Eric Z. Beard

Eric Z. Beard — Fri, 18 Sep 2009 12:59:27 +0000

I’ve been studying this issue closely for a while now. I have built and maintain a system with hundreds of millions of records that’s growing at about 25% per quarter consistently, so I’m obsessed with data storage. My application relies on hundreds of queries that need to run in real-time against all of that transactional data – no offline cubes or Hadoop clusters. I’m considering a jump to NoSql, but the lack of ad-hoc queries against live data is just a killer. I write probably a dozen ad-hoc queries a week to resolve support issues, and they normally need to run “right now!” I might be analyzing tens of millions of records in several different tables or fixing some field that got corrupted by a bug in the software. How do you do that with a NoSql system?

My solution for the moment is sharded databases, with a healthy dose of de-normalized aggregates sprinkled throughout the model, which are kept in synch with triggers or controlled data access via stored procedures. There’s no chance of data being different in two places.

And there’s no edict on how to write queries – if you join 20 tables and it runs fast, fine. If you need to write parallelized queries to return raw data from one table at a time and then join them on an application server, then that’s fine too.

I think a lot of people new to this game don’t get how good a SQL database can be. On a single shard (a cheap 2U box), I can run 5 million customers, which is something like 150Gb of data, and everything runs fast enough to keep people happy. In a NoSql system, how many nodes would you need to support both transactional and reporting applications for that much data?

By: Rayk

Rayk — Fri, 18 Sep 2009 09:03:43 +0000

Again a beautiful and nice posting, beside the content, which is very good!

For ad-hoc reporting we BI experts are currently facing the arising of coulumn based and column indexed storages, mainly in combination with in-memory technology.

As developer one does data modelling in his / her best way, in preparation for access things are split into columns and links between. Far away from 3NF or documents.

I supose it’s worth having a look at during your further investigations. ‘am looking forward.

Regarsd
Rayk

By: stephan

stephan — Fri, 18 Sep 2009 05:08:45 +0000

@Curt: But materialized views are complex writes – done by the DB – to denormalize data. The same I write about in the post. Aren’t they?

By: Curt Adams

Curt Adams — Thu, 17 Sep 2009 22:54:49 +0000

My experience with non-normalized data is that it will become inconsistent, guaranteed. You have to have a normalized backbone for important data. Complex views of normalized data are very useful and I’ve been using a materialized view strategy for 25 years, long before I ever heard of any formal descriptions. I have never been able to get any other system to work.

In terms of chunking, I find extremely complex “materialized views” are very useful. I haven’t used the Oracle system, but I’ve written some pretty complicated update routines that I would be surprised to see in standardized packages. So I think your goals can be best realized (in most environments) with sophisticated systems comparable to materialized views – documents that update automatically based on an underlying verifiable data system.

By: stephan

stephan — Tue, 15 Sep 2009 20:46:46 +0000

@Yeroc: Yes, you are right.

“I think it’s worth mentioning that some SQL databases (Oracle for example) provide out-of-the-box solutions for this problem as well and arguably in a more seamless manner. For example, creating a materialized view based on a complex query gives you the benefits of a denormalized schema.”

There are other benefits of chunks in documents and values though. Distribution comes to mind, which is inherently more difficult (might I say impossible above certain levels) to achieve with Oracle. And of course you can have denormalized data in every SQL database.

By: Yeroc

Yeroc — Tue, 15 Sep 2009 20:23:00 +0000

I think it’s worth mentioning that some SQL databases (Oracle for example) provide out-of-the-box solutions for this problem as well and arguably in a more seamless manner. For example, creating a materialized view based on a complex query gives you the benefits of a denormalized schema. This, in conjunction with query re-writing could mean your application would continue to operate normally without any knowledge of the denormalization going on behind the scenes. Meanwhile, at the application level inserts and updates don’t need to be concerned about manually updating the data across multiple tables since that will be handled automatically via the materialized view mechanism.

By: Rusty Wright

Rusty Wright — Tue, 15 Sep 2009 16:48:37 +0000

A good example of where this played out was Twitter, back when it kept falling on its face because it was using an sql database for storing everything.

By: Michael Mahemoff

Michael Mahemoff — Tue, 15 Sep 2009 15:52:24 +0000

+1 this is a great high-level summary of the architectures. The characterisation of a simple trade-off is a good starting point.

By: stephan

stephan — Tue, 15 Sep 2009 15:51:16 +0000

@Joe: Yes, this might be the case under some circumstances and some applications. Those were certainly the mainstream in the 80s and 90s. Internal applications where 1000 concurrent users (flight terminals?) are a high number.

But they only scale with massive caching. Today many applications experiences millions of concurrent users. Then you need to trade consistency with scalability.

As I’ve said to Stefan, the downside is “mainly the lack of adhoc reporting and adhoc data fixing.”. Which sometimes is a major issue.

So for user data, payment data, orders it’s still a good idea to keep your data normalized.

Cheers
Stpehan

By: Joe

Joe — Tue, 15 Sep 2009 15:42:16 +0000

In applications I have worked on, data consistency has been the highest concern. The performance loss to joins has been minimal in almost every case, and the ability to change\add data without having to remember to do it in 10 different tables is essential.

I will likely stick to Normalized Data for the majority of my applications, but I will have to keep an open mind towards DeNormalized data as well, I suppose.

By: stephan

stephan — Tue, 15 Sep 2009 15:33:47 +0000

@Stefan: Thanks. Well yes, there are many other tradeoffs. I’m writing a follow up post about the “Dark side of NoSQL” – mainly the lack of adhoc reporting and adhoc data fixing.

Cheers
Stephan

PS: And many more things

By: Stefan Schubert

Stefan Schubert — Tue, 15 Sep 2009 13:40:33 +0000

Hey Stephan,

very neat and clean presentation of the trade-off those two principles.
Unfortunately there is more involved in such a decision, but hopefully you’ll get into that later ^^

Chears
Stefan