Yesterday at the GigaOM Structure Big Data conference in New York City, Apache Cassandra parent company DataStax announced a new product that integrates Cassandra, Hive and Apache Hadoop. The new product, called Brisk, essentially uses Cassandra to replace Hbase and the Hadoop storage layer. “There’s a lot of nice properties about using the Hadoop programming model on top of a Cassandra layer, especially if you’re already using the database and want to do more large-scale batch processing,” says ReadWriteWeb resident big data expert Pete Warden.
This new permutation of Hadoop represents the ongoing evolution of the database.
A Cambrian Explosion of new programming languages, frameworks and databases has been underway in recent years. RedMonk’s Stephen O’Grady attributes this to a shift in preference from general purpose tools to specialized tools. Relational database systems are general purpose tools that have ruled for decades. Now non-relational databases are showing up for many special purposes, and new methods of interacting with datastores – such as Hadoop – are emerging to deal with the data.
What we’re seeing now is not necessarily a consolidation in the market, though that’s happening in the columnar database space and elsewhere. More importantly we’re seeing technologies converge, as projects split apart and combine to form new tools. The process is already underway, as Cassandra itself combines concepts from Amazon.com’s Dynamo and Google’s BigTable.
The merger between CouchOne and Membase is another example. This was both a market and technical consolidation: the forthcoming Couchbase Elastic Server will combine Membase and Apache CouchDB.
Hadoop and Hbase have been around for a few years now, and Google has been using MapReduce and BigTable for even longer. But the relational database paradigm has been around for about four decades. These new databases, and the algorithms for manipulating them, are still remarkably new.
Projects like Brisk and Couchbase are likely only the beginning of the forks, mergers and strange offshoots we’ll see in coming years.
