Join is a logical operation.

On my Texas Instruments scientifica calculator there is a noticeable pause when calculating powers. Does this mean I should stop using power?

Particular implementations of join might be slow, but to talk about join itself being slow is as absurd as saying power is slow.

I never even thought about the fact that this might happen some day.

I think we’re at least 10 years away from any one particular implementation of that idea gaining widespread use but the fact that it’s coming is interesting.

Thanks for the blog post :)

One of the downsides that you haven’t mentioned is maintenance (admin)

I worked on a project in which we decided to use an LDAP directory to store the user accounts, some content on the filesystem, and the bulk of the data in an RDBMS.
In theory it was a good idea – each storage system aligned to the job it is designed for.

In practice, the directory has been troublesome, and we wouldn’t do it again.

Not because our directory server lacked features, but because it’s a specialised tool for a specialised task, and that requires specialised skills to maintain it.

We have a storage team that manage our filesystems, SAN infrastructure and file backups, and it all runs smoothly.
We have DBAs who manage our RDBMSs, and maintain backup schedules, check free space, and run integrity checks.

What we didn’t have was someone with the knowledge and responsibility to do that for a more specialised storage system. We could have hired someone. We could have trained someone. We actually would have needed multiple people in order to ensure we had support 365 days a year.
We also needed to train our developers and operations staff.
We needed to build integration from our site-wide security manager into the directory in order to manage admin passwords and permissions.

For a system that only provided marginal improvement over a relational database, the overhead of all that was too high.

In retrospect we were further down the “dangerous” end of the spectrum that we realised. In retrospect, putting in a specialised storage system for 1 part of the application had a negative ROI.

(For the record, we don’t regret putting things on the filesystem)

Some applications need specialised stores. Some applications are big enough that the teams running them can justify hiring a team of CouchDB admins.
Some applications *need* full text search, so you’re going to have to make Lucene/Solr work – Your RDBMS isn’t going to get the job done.

But for a lot of cases, it’s much like the polyglot programming scenario – a 10% performance improvement (on paper) isn’t a strong enough benefit to justify the investment you’ll need to get it running seamlessly. For the money that you would need to spend, you can probably solve it more easily with hardware.

Developers would be much more effective if they more quickly embraced the right language for the right job with team buy-in. A team I’m on has already seen great benefits from attaching a scripting language to our Java runtimes to manually manipulate objects in the runtime context — absolutely invaluable! If we switched to a functional language we could also trim out about 30% Java-boiler-plate code, but the team isn’t ready for that yet.

Finally, there are just as many pitfalls with using a storage engine your team is not intimately familiar with! If we ever meet in a coffee shop I’ll have to give you the long list of Solr war stories. :)

Thanks for the post!

Many companies have been using diverse storages for years (Facebook, Flickr, Amazon, …) and others have followed with NoSQL in production environments (Reddit, Digg).

“Eventually some degree of shakeout will be required.”

Will happen for sure.

“I think #nosql is an extremely promising area with great potential.”

#NoSQL is only part of polyglot persistence.

a. More robustness and confidence building : Storage systems need to exhibit confidence. Thats one reason I think people flock to Oracle – that you will not lose data. And I think more production success stories need to emerge before such confidence can start becoming contagious.

b. IMO there’s way too much specialisation happening in this momentum. Eventually some degree of shakeout will be required. Thats still early work in progress I would imagine.

I do believe that going forward people will want to see some migration paths across such databases as a mechanism to protect their investments. That may slightly take the edge off the amount of polyglotism in the space, probably more towards some degree of standardisation.

I think #nosql is an extremely promising area with great potential. I believe the pace at which I imagine this potential will be converted into solutions and acceptance may not be as high as many of us might anticipate.