Take, for example, the new book from data experts Viktor Mayer-Schonberger (University of Oxford) and Kenneth Cukier (The Economist), Big Data: A Revolution That Will Transform How We Live, Work and Think. Mayer-Schonberger and Cukier urge us to trust data, not worrying about trying to understand correlations but simply to accept it. As Cukier tells Wired, "Big Data enables us not to test [a] hypothesis, but to let the data speak and tell us what hypothesis is best. And in that way it completely reshapes what we call the scientific method or...how we understand and make sense of the world."

One big problem with this view is that it assumes we have any clue how to query the data to even come up with a "what," much less a "why." It's not as if data simply presents itself to us, and we read it objectively.

Quoting Silver at length:

"[Big Data] is sometimes seen as a cure-all, as computers were in the 1970s. Chris Anderson…wrote in 2008 that the sheer volume of data would obviate the need for theory, and even the scientific method….

"[T]hese views are badly mistaken. The numbers have no way of speaking for themselves. We speak for them. We imbue them with meaning….[W]e may construe them in self-serving ways that are detached from their objective reality.

"Data-driven predictions can succeed--and they can fail. It is when we deny our role in the process that the odds of failure rise. Before we demand more of our data, we need to demand more of ourselves….Unless we work actively to become aware of the biases we introduce, the returns to additional information may be minimal--or diminishing."

So, for example, more data has not resulted in less political divide, as Silver points out. It has only hardened positions on either side of the aisle. The same holds true for global warming science. The more data we have, the less we seem to agree.

Why? Because data is never neutral. Or, rather, our perception of it is not neutral.

This is as true for individual enterprises grappling with product or personnel decisions as it is for countries debating policy issues. Big Data can contribute to the solving these issues...even as it contributes to making them more difficult. Again quoting Silver:

If the quantity of information is increasing by 2.5 quintillion bytes per day, the amount of useful information almost certainly isn't. Most of it is just noise, and the noise is increasing faster than the signal. There are so many hypotheses to test, so many data sets to mine--but a relatively constant amount of objective truth.

This jibes with Gartner's Svetlana Sicular, who suggests that "Formulating a right question is always hard, but with big data, it is an order of magnitude harder," due in part to the difficulty of figuring out meaningful correlations in our data. 

Again, while it may seem convenient to wish for the "data to speak for itself," it simply doesn't. It can't. It is always mediated by imperfect individuals with all of our biases, strengths and self-interest.

Which is not to say that data can't help us with our answers. Silver certainly turns to data to help him forecast elections, baseball games and Oscar winners. The trick, as he argues, is to take a Bayesian approach to data analytics, getting comfortable with probabilities, working hard to recognize and account for our biases, and not trying to predict certainties. When we predict certainties, we are almost always wrong.

In short, Big Data has tended to come with its share of Big Hype. So long as we're realistic about its potential, and recognize that our data is only as useful as the human intelligence we bring to it, minus the human biases with which we burden it, Big Data should, indeed, pay significant dividends.

But Silver was right and the pundits were wrong - and the impact of this goes way beyond politics.

Silver won because, um, science. As ReadWrite's own Dan Rowinski noted,  Silver's methodology is all based on data. He "takes deep data sets and applies logical analytical methods" to them. It's all just numbers.

Silver runs a blog called FiveThirtyEight, which is licensed by the Times. In 2008 he called the presidential election with incredible accuracy, getting 49 out of 50 states right. But this year he rolled a perfect score, 50 out of 50, even nailing the margins in many cases. His uncanny accuracy on this year's election represents what Rowinski calls a victory of "logic over punditry."

In fact it's bigger than that. Bear in mind that before turning his attention to politics in 2007 and 2008, Silver was using computer models to make predictions about baseball. What does it mean when some punk kid baseball nerd can just wade into politics and start kicking butt on all these long-time "experts" who have spent their entire lives covering politics?

It means something big is happening.

Man Versus Machine

This is about the triumph of machines and software over gut instinct. 

The age of voodoo is over. The era of talking about something as a "dark art" is done. In a world with big computers and big data, there are no dark arts.

And thank God for that. One by one, computers and the people who know how to use them are knocking off these crazy notions about gut instinct and intuition that humans like to cling to. For far too long we've applied this kind of fuzzy thinking to everything, from silly stuff like sports to important stuff like medicine.

Someday, and I hope it's soon, we will enter the age of intelligent machines, when true artificial intellgence becomes a reality, and when we look back on the late 20th and early 21st century it will seem medieval in its simplicity and reliance on superstition.

What most amazes me is the backlash and freak-out that occurs every time some "dark art" gets knocked over in a particular domain. Watch Moneyball (or read the book) and you'll see the old guard (in that case, baseball scouts) grow furious as they realize that computers can do their job better than they can. (Of course it's not computers; it's people who know how to use computers.)

We saw the same thing when IBM's Deep Blue defeated Garry Kasparov in 1997. We saw it when Watson beat humans at Jeopardy.

It's happening in advertising, which used to be a dark art but is increasingly a computer-driven numbers game. It's also happening in my business, the news media, prompting the same kind of furor as happened with the baseball scouts in Moneyball.

Who wants to believe that machines can tell us which stories to write, or which stories people want to read? Who wants to believe that machines can actually write stories? But they do. Forbes, my former home, started running computer-generated stories earlier this year. 

Backlash

Each time this happens, there's always lots of sputtering and outrage. When you strip away the rhetoric, at the core is always the same fear - that machines will take away jobs from humans. Baseball scouts want to keep working. So do journalists and chess masters.

So do pundits, but after this election it's getting harder to see what role they should play.

Why listen to Joe Scarborough and his crew on MSNBC's Morning Joe show bloviate about who's going to win and why, when Nate Silver and his computers can just give you the correct answer?

Scarborough bet Silver $2,000 on the outcome of the election after accusing Silver of being an "ideologue" who was predicting an Obama win simply because that's what Silver wanted to happen.

Scarborough has lost a lot more than the two grand. He's lost his reason for being, and he has taken a lot of others down with him.

Scarborough, not Siliver, turns out to be the wishful-thinking ideologue. Nate Silver and his computers may not put Scarborough and his ilk out of business - there's loads of airtime to fill, and windbags are still needed for that.

But Silver has exposed those guys for what they are, which is propagandists and entertainers.

And that's fine. We still need entertainers. Computers haven't learned to do that yet.

For now, anyway. 

Lead image courtesy of Shutterstock. Nate Silver image by Randy Stewart.