Big Data: Overhyped And Overpaid?

Gartner research director Svetlana Sicular thinks Big Data is about to plummet off the "peak of inflated expectations" into the "trough of disillusionment." Perhaps. But other data from Twitter and job trends suggest a much more complicated picture.

Sicular reaches her conclusions about Big Data based on a series of conversations with IT professionals over the past few weeks, in addition to a roundtable with Hadoop vendors Cloudera, Hortonworks, and MapR. In discussing Hadoop, the vendors suggest that "MapReduce has always been Hadoop’s bottleneck or that Hadoop is 'primitive and old-fashioned,'" apparently disillusioned with the state of Big Data's poster child/elephant. 

This could be chalked up to the Hadoop vendors simply acknowledging that despite being an excellent technology, Hadoop still has a ways to go. But Sicular's conversations with enterprise business analysts are more damaging:

My most advanced... Hadoop clients are also getting disillusioned. They do not realize that they are ahead of others and think that someone else is successful while they are struggling. These organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions... Formulating a right question is always hard, but with big data, it is an order of magnitude harder, because you are blazing the trail (not grazing on the green field).

And yet, these same companies don't seem to be giving up on Big Data. 

For example, DataSift plowed through 2.2 million Twitter mentions by more than 981,000 authors, as Ovum analyst Tony Baer reports, finding that positive mentions of Big Data vendors outnumber negative mentions by 3-to-1. And while Baer acknowledges that "Twitter streams are not a scientific focus group for detecting brand awareness, they provide a valuable window on market thinking." Indeed, given the levels of Big Data hype, it's surprising that the overall mood about Big Data remains overwhelmingly positive.

So much so, in fact, that enterprises are paying a premium to hire job candidates with Big Data-relevant technology skills, as Dice.com's 2012-2013 annual salary survey reveals. Job candidates with Big Data technology expertise command an average salary of $100,000, while other hot technologies like cloud/virtualization ($90,000) and mobile ($80,000) yield lower salaries. As Alice Hill, managing director of Dice.com, asserts, "We’ve heard [Big Data] is a fad, heard it’s hyped and heard it’s fleeting, yet it’s clear that data professionals are in demand and well paid."

While Gartner clearly has a valid point that Big Data's outsized expectations are sure to crash into reality at some point, it's also clear from jobs data, in particular, that enterprises see enough value from their data that they're willing to pay up for expertise that can analyze it. Will they be disappointed? Possibly. But the jobs data indicates we have yet to plummet into Gartner's "trough of disillusionment."