One of the most astoundingly sudden changes ever to happen in the software landscape in so short a period of time is the rise of “big data” and NoSQL, and the companies that manage it. What started out as a laboratory side-burner project at Yahoo just last year, is today a fully-funded company, with Hortonworks literally having managed to graft Hadoop onto a future Windows Server.

Breaking structured data out of its proprietary molds has enabled a completely new class of databases to lead the migration of enterprises to cloud storage platforms, leapfrogging from zero to hero. In the dust – it would seem – are companies like Oracle, SAP, and Sybase that suddenly find themselves playing catch-up. 2012 should be a superb growth year for Cloudera, Hortonworks, MapR, and the many new companies and organizations riding the open source, big data growth wave. And they should enjoy the party while it lasts, says Dan Vesset, IDC’s program vice president for data warehousing and analytics, because history tells us such companies only grow so far.

“We’re talking about tiny, tiny players with a few million dollars in professional services revenue,” Vesset tells RWW, “against multi-billion-dollar corporations. Any time that Oracle or IBM or SAP decide to get in, Hortonworks is just going to disappear. And I suspect Cloudera, MapR will [also] be acquired.”

Attrition! Attrition! (as sung by Zero Mostel)

Enjoy these brands while you can, the veteran IDC analyst believes, because before too long they’ll become rolled into the portfolios of major players. A decade ago, when Vesset was analyzing the then-emerging field of data warehouse appliances, “we had at least a dozen vendors come in within a very short time period, three years.” For example, does anybody remember DATAllegro, which blazed extraordinary new trails in data warehousing technology, to become acquired by Microsoft in July 2008? IBM acquired DataMirror; EMC acquired Greenplum; HP acquired Vertica. These were all data warehousing companies that were blazing trails just three years ago; already, they’re distant memories.

Vesset believes the big vendors will give Hortonworks and its brethren “leg room… for a few years.” They’ll rise up in the oven, and once they’re golden brown and baked just right, they’ll get consumed. “Maybe one of them will be able to get out organically and create a market,” he concedes, if it heeds the lessons of Red Hat and establishes a workable business model for itself around support and service.

And here’s where a new, big data company looking to survive rather than become acquired, might find an opportunity. Vesset cites a continuing demand for real-world tools to manage Hadoop-style data – tools that feel more like platforms than screwdrivers, and that don’t have names like Pig and Sqoop.

“Big vendors are not going to use open source technologies to put anything into production. It’s just not feasible for them,” he tells us. “They’re looking for support functions from companies like Hortonworks or Cloudera or EMC… additional automation software, additional tools and applications to help make Hadoop distributions more manageable by the average administrator or developer, so you don’t have to have high-end Java experts.”

Horton who?

The customer perception studies that Vesset reads tell him that a majority of information professionals today are unaware of Hortonworks’ existence. Some of his colleagues who attended the Hadoop World conference in New York last month reported the show was certainly newsworthy, although it was led by “a bunch of excited engineers looking around for a problem to solve.” For him, that’s the first sign of history repeating itself: When the data warehousing market took off 15 years ago, he relates, “there were a lot of failed projects because IT took the lead, built up very elegant, very sophisticated systems, and then wondered why nobody’s using them.”

The vendors Vesset believes will eventually acquire the big data organizations will be searching for methods of integrating Hadoop with their traditional data management technologies – methods that may not have been released yet by that time. Will we come to know what’s left of Hadoop using names like “Oracle Enterprise Application Cloud Adaptation Manager 2014?”

“I think within three or four years, more of the large vendors will be seeing real revenue opportunities in this market,” he responds. “So we will see more of the big vendors providing some type of solutions and services. Not to say within three or four years that all the startups will be gone; I think there will be a little bit of both.”

But hasn’t it always been the case that small startups apply themselves to narrow use cases and limited markets, becoming the king of their own respective hills, only to find themselves acquired by larger vendors that see the potential in leading those little fiefdoms… only to forget they exist within a few years, or sooner?

“In any given period of time, [there are] great opportunities for startups. Longer-term, things tend to aggregate. That’s not just IT; it’s the automotive industry, hotels, airlines. There are a few large vendors in control of most of the market, but there are always smaller vendors who are doing something interesting.”