Amazon Web Services is on fire, and EMC and VMware are feeling the heat. So the established enterprise-computing duo is striking back — by launching Pivotal, a joint venture that aims specifically to dethrone the current king of cloud computing.

Pivotal is led by Paul Maritz, the ex-CEO of VMware and a former senior executive at Microsoft. In leading the charge against AWS, Maritz is diving into a cloud-computing mosh pit that will include other tech heavyweights, such as IBM, Microsoft and Oracle.

(See also: Amazon: Can It Stay King Of Cloud Computing Forever?)

Pivotal heads for battle with parent-company assets — database technologies, data analytics and an application platform — it is combining into services that customers can lease to run their own software in the cloud. EMC owns 69% of Pivotal and VMware the rest. The two owners will have to invest a total of $800 million this year and next in order to kick start Pivotal, which Maritz conservatively estimates will reach $1 billion in revenue in five years from $300 million this year.

Amazon's Lead

Those numbers show how long it will take Pivotal to catch up with AWS. While Amazon won't break out the numbers for its cloud-computing unit, analysts say it is lumped inside the revenue category the online retailer calls "other." In Amazon's fourth quarter earnings released in January, "other" accounted for $769 million in revenue for the quarter and $2.52 billion for the year. That's a respective growth of 68% and 64%, according to the International Business Times. 

(See also: VMWare: "If Amazon Wins, We All Lose")

Nevertheless, the market is still young. Most AWS customers today are startups and small and medium-sized businesses. Amazon is expected to shift focus to large companies soon, heading right into EMC's and VMware's sweet spot. This is making both companies very nervous.

During a partner conference in February, VMware Chief Executive Pat Gelsinger warned that if "a workload goes to Amazon, you lose, and we have lost forever," CRN reported. To avoid that kind of customer drain, Pivotal will provide the public-cloud option for VMware customers using its infrastructure technology for private clouds. Supporting that migration is important to EMC, because it owns 80% of VMware.

Pivotal In The Cloud

On paper, Pivotal will provide an enterprise-class cloud-computing platform and infrastructure. The company includes Greenplum, EMC's Big Data analytics division, and Pivotal Labs, the storage company's application development environment. VMware is contributing cloud-computing platform CloudFoundry, and middleware and tools for building and running data-intensive Java applications.

Maritz will have to build a business on top of all this technology, but EMC's and VMware's commitment to Pivotal shows how they believe customer migration to cloud-computing environments outside their data centers is inevitable. The companies also know that failing to have what customers want would be suicidal.

In 2011, Gelsinger, then president and chief operating officer for EMC, said the company did not intend to become a casualty of any major change in the industry.

"The technology industry is ruthless and relentless," he said during an interview at the VMworld conference. "If you are not in front of those major waves of technological innovation, you will become one of the driftwood on the shores of the industry."

In cloud computing, stopping Amazon is how EMC and VMware plan to reach that shore alive.

Image courtesy of Shutterstock

The Hadoop market is getting interesting. Last year it was a death match between startups vying to own the heart of the project. Today it's a veritable smorgasbord of big-brand vendors getting involved to ensure they claim a big piece of the Big Data pie. Unlike American youth athletics, not everyone will get to take home a trophy.

Hadoop plays a key role in the burgeoning Big Data market, and represents a $13 billion market by 2017, according to Markets and Markets. (IDC pegs the market much, much lower at $812.8 million in 2016, but its numbers don't seem credible to me as they don't even seem to include Cloudera's sales.) Given that Big Data is hot, and Hadoop's data processing engine sits at its core, there's going to be a lot of money trading hands for Hadoop-related products and services.

Not everyone is going to collect.

SiliconAngle's John Furrier has challenged me on this, arguing that Hadoop is "not a winner take all market." While I, too, can see multiple winners in Hadoop, just as there have been in Linux (e.g., Red Hat dominates license/services revenue, but IBM, HP, and others make arguably more with related hardware, complementary software products, and professional services), markets don't tend toward entropy. They trend toward consolidation.

Today, the Hadoop ecosystem increasingly represents entropy:

• Cloudera, Hortonworks, and MapR remain the early favorites, but with very different approaches. Hortonworks positions itself as the 100% open source player; Cloudera somewhat does the same, but adds in complementary, proprietary bits, mostly around managing Hadoop, to add value to Hadoop (and its top line revenue); and MapR provides a hybrid open source/proprietary Hadoop distribution that swaps out HDFS for its proprietary NFS storage layer.
• EMC Greenplum has been involved with Hadoop for several years, and is set to release a new distribution of Hadoop called Pivotal HD. I've labeled Pivotal HD proprietary, but EMC's Hadoop team has taken issue with this characterization, arguing that PivotalHD is 100% open source, with complementary functionality (like HAWQ) available as add-ons. Point well taken, and I apologize for my misunderstanding. I was wrong, perhaps not surprisingly getting confused by Pivotal HD's product page, which says little about open source. But what seems clear is that customers won't be confused by EMC's value proposition: Hadoop with an advanced SQL query engine to make it easier and more powerful to use.
• Intel just got into the game with its own Hadoop distribution. Basically, you can think of it as Hadoop on (Intel Xeon™ processor, Intel SSD, and Intel 10GbE networking.hardware) steroids.
• For those who don't want to run Hadoop within the datacenter, Amazon offers Amazon Elastic MapReduce (EMR). As of April 2012, EMR was powering over 1 million Hadoop clusters. Presumably this number is much bigger today.
• Many, many others including IBM BigInsights, a range of startups, and more.

Will all of these companies make serious bank on Hadoop? No. Will some of them? Sure.

Ultimately, the winners in Hadoop will be those that invest most heavily in its success, as they will be perceived as the companies best positioned to help would-be customers succeed with Hadoop's complexities. But how they invest is up for discussion. Code to Apache Hadoop? Value-adding extensions?

Success isn't about open source purity, as Gartner's Merv Adrian posits: it's about making customers successful. As we saw with Linux, where Red Hat is both the top contributor to the Linux kernel and the company that harvests the most revenue from distributing Linux, contributing code is a great way to signal to the market that you're a leader and capable of getting code fixes to support customers. Code matters.

But code contributions are not the only way to demonstrate leadership and attract customers. Ultimately, companies that make it easier to get value from Hadoop will win big. There may be more than one such company. Indeed, there almost certainly will be. 

But there won't be 20 of them. Or even 10. Enterprise IT is simply not going to be able to manage a polyglot Hadoop distribution ecosystem. That's not the way markets work. No one wants to be "long tail" vendor, and customers don't want to buy from them, either, as Hugh MacLeod humorously points out on Gaping Void:

Source: GapingVoidArt. Used with permission.

The Hadoop market over the next year is going to be hugely interesting. And bloody.

Image courtesy of Ehab Othman / Shutterstock.

Hadoop, nearly synonymous with Big Data, has many failings. But open source is not one of them. In fact, Hadoop's open-source license remains one of its biggest draws, giving enterprises plenty of reasons to persevere in using it despite its shortcomings. It's therefore hard to see how EMC's new Pivotal HD, essentially a proprietary distribution of Hadoop, can hope to succeed.

Not that everyone agrees with this statement.

Dan Woods, CTO and editor of CITO Research and a contributor to Forbes, argues that embedding Hadoop into EMC Greenplum's massively parallel processing (MPP) database (HAWQ) offers CIOs and CTOs the simplicity they need to be successful with Hadoop. He has a point: Hadoop is complex and somewhat hard to use, which is why Cloudera CEO Mike Olson has argued that most of the world will experience the power of Hadoop through applications, nearly all of which will be proprietary, I might add.

But Olson's argument differs from Woods' argument in at at least one major way: Pivotal HD is enterprise infrastructure, not an application, and enterprise infrastructure is increasingly open source.

There are plenty of reasons for this, but RethinkDB's Alex Popescu nails one critical factor:

Hadoop is so successful despite its complexity [because i]t allows experimenting and trying out new ideas, while continuing to accumulate and storing your data. It removes the pressure from the developers. That’s agility. It’s highly appreciated.

In other words, a big reason for Hadoop's success is its open-source license, which permits a hefty amount of experimentation without having to get an enterprise license from EMC, Oracle, or any of the other incumbent infrastructure vendors.  

EMC's Scott Yara tries to deflect criticism of its proprietary foray into Hadoop by declaring "We're all in on Hadoop, period," but as 451 Research analyst Matt Aslett counters, "I have no doubt that EMC Greenplum is 'all in' on Pivotal HD, but that’s not the same thing at all."

Take this away by building a proprietary Hadoop distribution, and EMC has basically erased the very thing that made Hadoop workloads proliferate in the first place. EMC also cuts itself out of the standard adoption cycle for Hadoop, as Redmonk analyst Stephen O'Grady suggests, "Certainly there will be customers whose needs will dictate the adoption of a unique solution like Pivotal HD, but how many will that be relative to the segment whose adoption cycle begins with the download of one of the free Hadoop distributions?"

Today, Hadoop is one of the industry's hottest job trends. Even in absolute job numbers, it's about to pass EMC-related job posts:

Hadoop,emc Job Trends

Hadoop jobs - EMC jobs

Enterprises aren't hiring for EMC's brand of Hadoop. They're hiring for the open source Hadoop. This matters.

Perhaps EMC feels that Hadoop's brand is big enough now that enterprises essentially understand it and are ready to move on from experimentation to full-scale adoption. In this EMC is likely to be disappointed. According to recent IBM survey data, only 6% of enterprises have two or more Big Data projects underway (likely, though not explicitly, involving Hadoop in some way), and a mere 22% are running pilots to test the efficacy of their Big Data strategies. Everyone else is in full-on planning mode.

By creating a proprietary Hadoop distribution, EMC just dramatically limited its access to the 94% that are still in Big Data education and trial mode. Yes, it has a gargantuan sales force. No, they're simply not going to be able to reach would-be customers as efficiently as an open-source distribution model does.

But maybe EMC hasn't gone proprietary to more effectively monetize Hadoop interest, and instead sincerely believes, like Woods ("open source development has its limits"), that complex infrastructure problems are a poor match for open source. History has not been kind to such thinking, as Aslett sarcastically implies:

"Enterprise products" always prevail over open source. onforb.es/VgQ0Cq That's why Linux has been such an abject failure versus Unix.

— Matt Aslett (@maslett) March 1, 2013

EMC has seemingly bottomless resources to throw at Hadoop, and every incentive to do so. It's a smart, highly successful company and no doubt will prove successful with Pivotal HD. However, I can't see it ever dominating an open-source infrastructure market with a proprietary distribution. Open source is the foundation for today's most interesting markets, from Big Data to mobile to cloud computing. It's unlikely that EMC will somehow stem this tide with a proprietary product, no matter its short-term performance or functionality advantages.

Guest author Ed Lee is lead architect for virtualized storage vendor Tintri.

Virtualization and flash memory are disrupting the staid storage industry. First embraced in slick consumer products like the iPod, flash memory is now the new darling of enterprise IT. One reason is speed.  Flash is more than 400 times faster than rotating disks.

The other reasons are virtualization and cloud computing. Combined, these technology trends have strained the capabilities of traditional storage products. Those big metal boxes of enterprise storage, called arrays, were originally designed in the 1980s - before MC Hammer rocked parachute pants. Today’s input/output storage requirements are heavier and far more random than storage designers could have predicted 30 years ago. And with new technology comes new problems. IT systems that once hummed, “Can’t Touch This,” became hung up by traffic jams. With the highly random I/O of modern virtualization, flash trumps even the best spinning disks.

Well, nothing gets venture capitalists writing checks faster than IT managers with large budgets wringing their hands in frustration. So, enter the flash solutions from enterprise storage startups. Even the incumbent dinosaurs are showing some hustle. Just last spring, storage industry giant EMC's dropped an impressive $400 million to snap up XtremIO, a three-year-old flash company that has yet to ring up a single sale.

Too Much Flash?

So has flash been over-hyped? Yes, of course it has. Most promising new technologies are overhyped, but the best survive the disappointment that is sure to follow. The problem with most of the flash crowd is that they tout it as a cure-all for IT's performance woes – like a magic medicine show, it puts the performance back in your data center. But it's not magic. It does one thing really well - it eliminates contention for disk spindles. But flash by itself does nothing to ease storage management burdens, and in fact may actually contribute to increased infrastructure complexity. 

Like all new technologies, flash-based storage systems need to be designed into complete solutions rather than just point products.   

Speed Isn't Enough

Today, flash is central to nearly every next-generation storage solution coming to market. But smart IT managers know that speed alone doesn’t solve all of their storage problems, especially in a world of virtualization and cloud computing. In fact, flash-based vendors that offer systems that simplify storage management provide a much greater boost to IT performance than just making I/Os go faster.

Like many incremental improvements in component technologies – even order of magnitude performance boosts like flash – are too often hijacked by legacy vendors to create incrementally smaller and faster versions of the same old products. Don’t be blinded by flash. It’s cool, but it needs to be integrated with new approaches to building complete solutions.

Expect the enterprise storage conversation, pushed by the new demands of virtualization and cloud computing, to move beyond flash. It’s not just about the speed; it's about the solution.

Image courtesy of Shutterstock.