Home Enterprises Take Off Their Hadoop Training Wheels

Enterprises Take Off Their Hadoop Training Wheels

The dirty little secret of Hadoop has been just how dull many of its tasks have been. By far the biggest use for Hadoop to date has been as a “poor person’s ETL“—that is, a form of data integration, at the risk of oversimplifying—rather than all the big, sexy data science we see constantly hyped.

But that’s changing. As a new Sand Hill Group survey reveals, a significant percentage of enterprises are moving beyond Hadoop’s mundane past to leverage it for advanced analytics.

Hadoop 101

This shouldn’t be too surprising. Hadoop is still new to most companies, with 47% of respondents in Sand Hill Group’s survey citing a lack of Hadoop skills, coupled with a shortage of talent to hire (21%) as top challenges inhibiting their Hadoop ambitions. It’s impossible to move from beginner to expert in the few seconds it takes to download Hadoop.

Compounding this problem, Hadoop has not traditionally been the most approachable system to use. Enterprises have been willing to muddle through its complexities, however, because it so dramatically lowers the cost profile of a Big Data project, given that it’s powerful open-source software running on commodity servers. While Hadoop is becoming easier to use, it still imposes a steep learning curve that requires time and experience to master.

So we’re seeing enterprises that started with Hadoop as their “unsupervised landfill” now moving to more complex, and important, workloads, as 451 Research analyst Matt Aslett points out.

Until companies feel confident in a given technology, they’re not going to start using it for mission critical applications. At one time it was anathema to use Linux in the data center, so it was used at the edge of the network for file and print servers and more pedestrian workloads. Now it would be anathema to not use Linux in mission-critical data center applications. 

Taking The Hadoop Training Wheels Off

We’re seeing the same thing with Hadoop. The majority of enterprises are still kicking Hadoop’s tires, running relatively small clusters of just five to nine nodes. But as this tire-kicking phase ends, the real work will be getting started.

While most every other workload will either decline or stay roughly constant going forward, Hadoop is expected to boom in advanced analytics workloads, as a Sand Hill Group survey showcases:

Even so, Hadoop remains somewhat mired in log data (61% of respondents are using Hadoop to store log data), followed by operational data used in CRM and ERP systems (53%). Very few companies are using it today for streaming, real-time data. Presumably this will change as enterprises look to put real-time data to use in advanced analytics projects. (More likely, they’ll do what Criteo and others are doing by marrying Hadoop’s advanced analytics capabilities with a NoSQL database for real-time data capture and response.)

Enterprises Are Betting Big On Hadoop

Regardless, what we’re seeing is a significant maturing of enterprise adoption of Hadoop. As the report notes:

This almost-threefold increase over the 8.9% currently developing advanced analytics is emblematic of the larger shift towards initiatives that can have a transformational impact on the organization. It also conveys both the aggressive expectations for skill and experience development and the urgent need to mine the available data to improve business decisions and results.

There will be hiccups along the way, of course, but as Hadoop enthusiast Floyd Strimling posits, “Hadoop bent the storage and compute cost curves to allow everyone to analyze data. There is no going back.”

Image courtesy of Shutterstock

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.