The allure of Big Data is that given enough data, human intuition stops mattering and the data takes over. It’s a beautiful utopia where, as The Economist‘s Kenneth Cukier avers, we can “let the data speak” and stop hunting for causation while the data reveals correlations.
Sadly, it’s complete nonsense.
In a way, we seem to know this already. In a variety of surveys, executives again and again insist that data is valuable … and then trust gut instinct, anyway. The key to overcoming this and welding data and intuition together may be to trust homegrown data analysts instead of outside data scientists.
We Love Data! (Sort Of)
Big Data is one of the most hyped phrases in technology today, and has led to big plans to harness it, according to Gartner research:
As much as we may proclaim our allegiance to Big Data, however, executives tend to ignore it in favor of their own intuition. Consider:
- 40% of CMOs make decisions based on intuition, not data (Formstack 2014)
- 37% of CMOs use data regularly when making decisions, despite roughly 50% claiming that data is their most underutilized asset (Formstack 2014)
-
62% of business executives said they tend to trust their gut, and 61% said real-world insight tops hard analytics when making decisions (Fortune Knowledge Group 2014)
This isn’t surprising, given the kind of data science most companies seem to employ. While most of the world’s “Big Data” is created by and for machines, most companies actually focus on the human side of Big Data, as a Gregory Piatetsky poll of his KDnuggets community illustrates:
In fact, as much as we may hope that data will somehow overcome our human failings, it’s our very humanity that gives data any chance of helping us at all.
People Are People
The problem is that we tend to assume that all data science is the same. It’s not. As former Google and Foursquare data scientist Michael Li makes clear, data scientists either produce analytics for machines or humans, but generally not both. The two types of analysis require different skills.
Li suggests that the ideal characteristics of someone doing data science for humans include the ability to package analysis in a human-understandable format, with an emphasis on storytelling and articulation of “how” and “why.”
The same is true of developers: as much as they may want to live in the land of ones and zeroes, the reality is that they spend far more time with people:
https://twitter.com/r343l/status/512361922780344320
Whether developers or data scientists or anyone else, then, the best people will be those that communicate well.
It’s therefore critical to find data scientists that speak the language of an organization, including its executives who are most likely to act on the data (or ignore it). Not surprisingly, it’s far easier to find such people within the organization rather than scouring the planet for a cure-all data scientist fresh off Wall Street or elsewhere.
The more familiar with a company’s business, the more likely the data scientist will be able to ask the right questions and articulate the answers.
Start Small, Go Big
Often there isn’t a single right person for the Big Data analytics job, but rather a team of people. As Michael Schrage, a research fellow at MIT Sloan School’s Center for Digital Business, highlights, a team approach may work best:
The smartest thing I’ve seen organizations start doing is seed-fund and empower small cross-functional data-oriented teams explicitly charged with delivering tangible and measurable data-driven benefits in relatively short periods of time. The accent is on the word team; the emphasis is on building greater data capability than better digital infrastructures. The goal is to make all of the organization—not just the geeks and quants—more conversant in how to align probability, statistics, technology and business value creation.
While this team approach may not obviate the need for external hires, Schrage suggests that it lays a solid foundation for external data scientists to build upon.
That foundation has become easier to build thanks to the cloud. As Matt Wood, Amazon Web Services’ data science chief, told me in an interview, the cloud has led to an “exponential curve” of Big Data talent growth. By teaming up and experimenting with Big Data in a flexible environment, we’re quickly building up data science expertise, usually in-house.
All of which suggests that the right way to get moving on Big Data within your organization may be to spin up small projects in the cloud, and make sure they’re staffed with a small, cross-disciplinary team within the organization.
Lead image courtesy of Shutterstock
