Home What Will This Bubble’s Legacy Be? Open Source Big Data and Analytics Tools

What Will This Bubble’s Legacy Be? Open Source Big Data and Analytics Tools

First of all, let’s leave aside the issue of whether we’re in a bubble or not, and just assume that we are. Ashlee Vance has an excellent piece in Business Week looking at one tragic aspect of this bubble: too many mathematicians are flocking to Silicon Valley to work for companies like Google, Facebook and Zynga to work on advertising platforms. Former Facebook employee and Cloudera co-founder Jeff Hammerbacher is quoted saying “The best minds of my generation are thinking about how to make people click ads. That sucks.”

I couldn’t agree more. But I disagree with the subheading of the piece “Tech bubbles happen, but we usually gain from the innovation left behind. This one–driven by social networking–could leave us empty-handed.” Thanks to this bubble, we’ve already got Apache Hadoop, Apache Cassandra, Membase and many other free open source tools for working with big data. If the bubble popped tomorrow, researchers in many fields would still have all of these tools.

Strangely, Vance doesn’t mention Hadoop once in the article – even though an enterprise distribution of Hadoop is the main product Cloudera sells. Hadoop is based on research papers published by Google, and Yahoo funded and continues to fund much of its development.

Facebook open-sourced its database system Cassandra in 2008. Zynga was one of the original contributors to Membase, which was released in 2010. Twitter open-sourced FlockDB. LinkedIn open-sourced Project Voldemort. VMWare sponsors Redis.

Venture backed startups such as Cloudera, DataStax and Couchbase have emerged to commercialize some of these projects, and these companies contribute back to the original projects.

And it’s not just run-off from Internet giants fueling this movement. Revolution Analytics, another venture funded startup, develops a free version of its distribution of the R statistical programming language. It recently announced that it would provide its enterprise edition for free for those competing in Kaggle’s data science competitions. Neo Technology open-sourced its graph database Neo4j, Basho open-sourced Riak and 10gen open-sourced MongoDB.

All these companies could go away. But the source code is forever.

And the innovation continues. Google published a paper on its graph analysis system Pregel, which has led to the open source projects GoldenOrb and Phoebus.

These are all tools that enable academic researchers, independent researchers with no funding, biotechnology startups and more. I think it’s a shame that so many minds are spending their time on advertising. And I think some of the valuations and investments in Silicon Valley are absurd. But I’m comforted by the fact that many of their tools are being released to the public.

Lead image by Umberto Salvagnin

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.