Hadoop plays a key role in the burgeoning Big Data market, and represents a $13 billion market by 2017, according to Markets and Markets. (IDC pegs the market much, much lower at $812.8 million in 2016, but its numbers don't seem credible to me as they don't even seem to include Cloudera's sales.) Given that Big Data is hot, and Hadoop's data processing engine sits at its core, there's going to be a lot of money trading hands for Hadoop-related products and services.

Not everyone is going to collect.

SiliconAngle's John Furrier has challenged me on this, arguing that Hadoop is "not a winner take all market." While I, too, can see multiple winners in Hadoop, just as there have been in Linux (e.g., Red Hat dominates license/services revenue, but IBM, HP, and others make arguably more with related hardware, complementary software products, and professional services), markets don't tend toward entropy. They trend toward consolidation.

Today, the Hadoop ecosystem increasingly represents entropy:

• Cloudera, Hortonworks, and MapR remain the early favorites, but with very different approaches. Hortonworks positions itself as the 100% open source player; Cloudera somewhat does the same, but adds in complementary, proprietary bits, mostly around managing Hadoop, to add value to Hadoop (and its top line revenue); and MapR provides a hybrid open source/proprietary Hadoop distribution that swaps out HDFS for its proprietary NFS storage layer.
• EMC Greenplum has been involved with Hadoop for several years, and is set to release a new distribution of Hadoop called Pivotal HD. I've labeled Pivotal HD proprietary, but EMC's Hadoop team has taken issue with this characterization, arguing that PivotalHD is 100% open source, with complementary functionality (like HAWQ) available as add-ons. Point well taken, and I apologize for my misunderstanding. I was wrong, perhaps not surprisingly getting confused by Pivotal HD's product page, which says little about open source. But what seems clear is that customers won't be confused by EMC's value proposition: Hadoop with an advanced SQL query engine to make it easier and more powerful to use.
• Intel just got into the game with its own Hadoop distribution. Basically, you can think of it as Hadoop on (Intel Xeon™ processor, Intel SSD, and Intel 10GbE networking.hardware) steroids.
• For those who don't want to run Hadoop within the datacenter, Amazon offers Amazon Elastic MapReduce (EMR). As of April 2012, EMR was powering over 1 million Hadoop clusters. Presumably this number is much bigger today.
• Many, many others including IBM BigInsights, a range of startups, and more.

Will all of these companies make serious bank on Hadoop? No. Will some of them? Sure.

Ultimately, the winners in Hadoop will be those that invest most heavily in its success, as they will be perceived as the companies best positioned to help would-be customers succeed with Hadoop's complexities. But how they invest is up for discussion. Code to Apache Hadoop? Value-adding extensions?

Success isn't about open source purity, as Gartner's Merv Adrian posits: it's about making customers successful. As we saw with Linux, where Red Hat is both the top contributor to the Linux kernel and the company that harvests the most revenue from distributing Linux, contributing code is a great way to signal to the market that you're a leader and capable of getting code fixes to support customers. Code matters.

But code contributions are not the only way to demonstrate leadership and attract customers. Ultimately, companies that make it easier to get value from Hadoop will win big. There may be more than one such company. Indeed, there almost certainly will be. 

But there won't be 20 of them. Or even 10. Enterprise IT is simply not going to be able to manage a polyglot Hadoop distribution ecosystem. That's not the way markets work. No one wants to be "long tail" vendor, and customers don't want to buy from them, either, as Hugh MacLeod humorously points out on Gaping Void:

Source: GapingVoidArt. Used with permission.

The Hadoop market over the next year is going to be hugely interesting. And bloody.

Image courtesy of Ehab Othman / Shutterstock.

The smartphone market is expected to grow. Again.

On Monday, IDC predicted that PC sales will fall 1.3% in 2013, and that smartphone sales will continue their explosive growth, topping 50% and displacing the legacy feature phone as the dominant mobile phone platform. 

Although IDC released the two reports separately, they're best considered together, for context. What IDC predicts merely reflects the conventional wisdom: that the age of the PC is ending, and that the smartphone is the dominant platform. And, if the Apple iWatch is real, and Google Glass becomes a viable platform, then we have the past, present, and future of the computing market: the PC, the phone, and wearable computing. 

The Windows 8 Phenomenon

The 2012 performance of the PC market could be written off as a consequence of Windows 8: the pause in sales before the launch, followed by what might be called a "mild" reception by the market. PC sales fell 3.7% for the year, IDC found, with an 8.3% drop in fourth-quarter shipments. U.S. PC sales fell 6.5% for the fourth quarter and 7.6% for the year.

"The PC market is still looking for updated models to gain traction and demonstrate sufficient appeal to drive growth in a very competitive market," said Loren Loverde, an analyst for IDC, in a statement. "Growth in emerging regions has slowed considerably, and we continue to see constrained PC demand as buyers favor other devices for their mobility and convenience features. We still don't see tablets (with limited local storage, file system, lesser focus on traditional productivity, etc.) as competitors to PCs – but they are winning consumer dollars with mobility and consumer appeal nevertheless."

Gartner hasn't yet released its 2013 PC forecasts, but has already said that PC sales dropped 4.9% in the fourth quarter, as it seems consumers just didn't really care about them any more.

Smartphones: A Worldwide Phenomenon

Smartphones, meanwhile, have worked their way through "mature" markets like the United States and into the high-volume, lucrative BRIC (Brazil, Russia, India, China) countries, IDC reports. As the smartphone begins selling in high volume in those regions, look for even higher shipment numbers: IDC predicts that more than 1.5 billion smartphones will be shipped by the end of 2017, worldwide, or more than two-thirds of the phone market. In India, for example, less than half of the phones sold there in 2017 will be smartphones, IDC predicted - and yet it will be the world' third-largest smartphone market.

Gartner, meanwhile, said that sales of mobile phones actually fell 1.7%  during 2012 - not because of lack of demand, but due to consumers turning to smartphones instead of feature phones.

Meanwhile, IDC reported earlier this month that tablet sales reached record levels, 52.5 million units, during the fourth quarter.

PC sales may yet rebound - Microsoft seems to believe that, and it still maintains close ties to enterprises and consumers. But, increasingly, the PC seems be a legacy device of interest to a slowly declining number of users.

While not generally expressed in this way, in my conversations with IT and line-of-business executives, such sentiment comes out in the subtext of what they say. The media (over)hypes Big Data and IT executives then come to think they must be doing something wrong, as Gartner analyst Svetlana Sicular has found in her conversations with clients. As the current thinking goes, if enterprises don't have warehouses overflowing with data, with data scientists madly crunching the data and coming up with "actionable insights," they're doing it wrong.

One Big Landfill

As just one example, I recently heard one executive at a Fortune 100 company say, "Hadoop is our unsupervised landfill." Spoken like a man who knows his data is important, but isn't quite sure why or how. So his company just stores everything in the hopes that all that data will one day make sense.

This is a reasonable response, given the pressures, but it's actually okay to not have The Big Data Answer. Odds are, your enterprise needs to figure things out over time, even without the mythical (and expensive) data scientists we keep reading about. Sicular argues that "Organizations already have people who know their own data better than mystical data scientists." Give these in-house experts the top tools for Big Data, described in a recent Dice.com job trends report, all of which happen to be open source, and let them iterate toward understanding the data. 

Indeed, open source is the key here, not how big your data is.  

Exploration, Not Exploitation

Alex Popescu nails it when he posits, "Hadoop is so successful despite its complexity [because i]t allows experimenting and trying out new ideas, while continuing to accumulate and storing your data." Unlike with proprietary technology, in open-source Big Data technology you don't have to sign any contracts, fork over any money, or do any of the things typically expected with enterprisee software vendors. You just download and explore.

This fact was underlined for me at a Big Data panel in Chicago this week, which featured Dr. Philip Shelley, CTO at Sears Holdings. Sears is arguably one of the industry's top pioneers when it comes to Big Data, and he insisted that open-source tools like Hadoop were critical to the company iterating its way to Big Data success. Things have gone so well that he has decommissioned millions of dollars in IBM Netezza and other proprietary technology to focus on Hadoop as its data hub. As he said, "We no longer have to budget for capital expenditures" for Big Data initiatives."

That's impressive.

Yes, data volume is growing. But that's cause for exploration and iteration, not frustration and despair, following Sears' example. You're not alone if you don't yet know what to do with all your data, or if you're wondering if you have enough to bother. As Brian Proffitt has pointed out, small companies with less than gargantuan data troves can also benefit from Big Data technologies, because "big" isn't really about size at all. It's also about variety and velocity of data, among other things.

Or, as Edd Dumbhill ably notes, "'Big data' really means 'smart use of data'."

That "smart use" will almost always involve open source, as explained above. But it should also involve the understanding that you're not in a race to amass data and to recruit data scientists to decipher it. Big Data is an iterative process of using (mostly) open-source technologies to store and analyze data in different ways, learning from peers and from your own experience. It needn't be a landfill of buzzwords.

Image courtesy of Shutterstock.

Sicular reaches her conclusions about Big Data based on a series of conversations with IT professionals over the past few weeks, in addition to a roundtable with Hadoop vendors Cloudera, Hortonworks, and MapR. In discussing Hadoop, the vendors suggest that "MapReduce has always been Hadoop’s bottleneck or that Hadoop is 'primitive and old-fashioned,'" apparently disillusioned with the state of Big Data's poster child/elephant. 

This could be chalked up to the Hadoop vendors simply acknowledging that despite being an excellent technology, Hadoop still has a ways to go. But Sicular's conversations with enterprise business analysts are more damaging:

My most advanced... Hadoop clients are also getting disillusioned. They do not realize that they are ahead of others and think that someone else is successful while they are struggling. These organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions... Formulating a right question is always hard, but with big data, it is an order of magnitude harder, because you are blazing the trail (not grazing on the green field).

And yet, these same companies don't seem to be giving up on Big Data. 

For example, DataSift plowed through 2.2 million Twitter mentions by more than 981,000 authors, as Ovum analyst Tony Baer reports, finding that positive mentions of Big Data vendors outnumber negative mentions by 3-to-1. And while Baer acknowledges that "Twitter streams are not a scientific focus group for detecting brand awareness, they provide a valuable window on market thinking." Indeed, given the levels of Big Data hype, it's surprising that the overall mood about Big Data remains overwhelmingly positive.

So much so, in fact, that enterprises are paying a premium to hire job candidates with Big Data-relevant technology skills, as Dice.com's 2012-2013 annual salary survey reveals. Job candidates with Big Data technology expertise command an average salary of $100,000, while other hot technologies like cloud/virtualization ($90,000) and mobile ($80,000) yield lower salaries. As Alice Hill, managing director of Dice.com, asserts, "We’ve heard [Big Data] is a fad, heard it’s hyped and heard it’s fleeting, yet it’s clear that data professionals are in demand and well paid."

While Gartner clearly has a valid point that Big Data's outsized expectations are sure to crash into reality at some point, it's also clear from jobs data, in particular, that enterprises see enough value from their data that they're willing to pay up for expertise that can analyze it. Will they be disappointed? Possibly. But the jobs data indicates we have yet to plummet into Gartner's "trough of disillusionment."