Home Big Data Failures Owe More To Business Culture Than Technology

Big Data Failures Owe More To Business Culture Than Technology

Hope springs eternal, especially in Big Data. Despite widespread failure to achieve much of anything with Big Data projects, gargantuan piles of cash keep flowing into such projects, hitting $31 billion in 2013 and expected to top $114 billion by 2018. 

Yet while 60% of executives believe Big Data will upend their industries within three years—according to a recent Capgemini report—a mere 8% describe their own projects as “very successful,” while another 27% call their efforts “successful.”

Given how much companies are spending, one would hope for better returns. Real success, however, derives from a cultural affinity for data, and not simply a technology purchase.

Failure All The Way Down

No one seems to dispute the inherent value of data, and the more of it the better. In Capgemini’s survey, fully 60% of respondents believe Big Data is going to change the world, starting with their industries. 

Yet when asked about the status of their Big Data initiatives, it’s clear that reality bites:

For those that have been paying attention, this isn’t really news. After all, Gartner found a few years ago that while everyone was jumping into Big Data, few knew how to make it work. 

As to why Big Data projects fail, the answer is “it depends.” Some of the reasons are cultural (“ineffective coordination of teams across the organization”), while others are more easily fixed (“dependency on legacy systems”):

Looking at this list, it’s hard to see how things improve in the short term. 

Teaching Old Data Dogs New Data Tricks

Some vendors pitch a “data lake” as the solution to the first problem. Capgemini’s survey finds that 79% of enterprises haven’t completely integrated data sources from across the organization. To make this simpler, the data lake advocates insist that enterprises don’t need to standardize data as it enters the organization; instead they can keep it in its original format and just store it in one big repository.

See also: Oh, Go Jump In A Data Lake

While this sounds simple, it’s not clear that it’s actually useful.

I’ve heard some call such a data lake a “Hadump,” an unflattering play on the decentralized storage framework Hadoop that suggests just because all the data resides in one place doesn’t make it useful. As Gartner analyst Nick Huedecker has pointed out:

The fundamental issue with the data lake is that it makes certain assumptions about the users of information. It assumes that users recognize or understand the contextual bias of how data is captured, that they know how to merge and reconcile different data sources without ‘a priori knowledge’ and that they understand the incomplete nature of datasets, regardless of structure.

So some fail with newfangled, Hadoop-inspired data lakes, while many more fail by trying to get antiquated data infrastructure (e.g., relational databases) to fit modern data (messy, disparate, and lots of it). 

The Cultural Problem

But the biggest cause of failure, even if not acknowledged as such, is that most enterprises simply don’t have a culture of data-centricity. At best, they treat “Big Data” as a discrete project with a definitive completion date.

As such, they’re not set up to succeed, as the Capgemini report finds:

There are many factors that go into the making of a successful Big Data implementation. However, the single biggest factor that we observed was that organizations that have a strong operating model stood apart. This operating model has multiple distinct elements, which include, among others, a well-defined organizational structure, systematic implementation plan, and strong leadership support.

Each of these three things ties into a corporate culture that appreciates and is built around data. 

I would also add, following something that Zoomdata’s Justin Langseth recently said to me, that design is an essential element of any successful Big Data project. The best Big Data projects will bring data to life for the rank-and-file within an enterprise, not merely the high priests and priestesses of data science. 

In sum, Big Data success flows from a cultural affinity for data, which can be sparked by a strong leader within an organization, but ultimately must become how an entire company thinks about its business.

Image courtesy of Shutterstock

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.