VMware has a firm if fading grip on the server virtualization market, but according to IDC analyst Al Gillen, virtualizaton serves as a convenient on-ramp to private cloud, which in turn leads to the public cloud. Is VMware paving IT's path to Amazon, Microsoft Azure, Rackspace and other public cloud providers?

Not exactly.

Virtualization: Still Relevant, Mostly VMware

According to Gillen, who spoke at the Open Business Conference (OSBC) in San Francisco earlier this week, VMware continues to dominate the virtualization market, with just under 60% market share. VMware's installed base, coupled with CIO resistance to change, mean that VMware's hold on virtualization should persist for years. 

That's the good news.

The bad news is that VMware faces fierce competition from Microsoft's Hyper-V, currently claiming over 25% of the market, as well as a strong and growing threat from KVM, now bolstered by a rising OpenStack. KVM deployments grew 50% last year, according to IDC. Xen, the other open-source virtualization alternative, remains robust but isn't growing as fast, though its move to the Linux Foundation may help to revive its growth.

By themselves, however, none of these virtualization competitors poses much near-term risk to VMware. Of far greater importance is a distinct trend toward multi-hypervisor environments, as well as an enterprise shift from virtualization to cloud.

Each of these trends threatens VMware.

Multi-Hypervisor Trend No Friend To VMware's Cloud

According to Gillen, some 15% of enterprises deploy multiple hypervisors today, but Gillen expects that number to double in the next one to two years, with cost being a primary driver for experimentation with new virtualization technologies. The more enterprises experiment with non-VMware virtualization technology, the more likely they will also diverge from VMware's cloud offerings.

Why? Because virtualization is a clear precursor to cloud adoption.

According to IDC's Platform Migration MCS, January 2012, roughly 80% of servers that enterprises are migrating to the cloud are already virtualized, rather than being virtualized as part of the migration. Often, enterprises will rely on their virtualization vendor to walk them into the cloud, with private clouds the first stopping point on the way to public clouds.

As such, VMware has actively been building out both private and public cloud options, creating a clear "upgrade" path for its enterprise buyers. As Matthew Lodge, VMware'e vice president of Cloud Services, emphasizes, VMware enables enterprises to stitch together “what they have in their data centers and their public cloud instance.” All running on VMware technology.

It's a compelling strategy, one also being adopted by Microsoft (Windows Server + Hyper-V + Azure) and Red Hat (Red Hat Enterprise Linux + Red Hat Enterprise Virtualization + OpenShift or OpenStack), among others. 

Virtualization Not The Only Path To The Cloud

But not all workloads follow this single vendor path. Indeed, Gillen cited IDC's 2012 Cloud System Software Survey, which found that transitions to the cloud allow vendors to "sell cloud system software on its own merits and embed a hypervisor as part of the package." Some 53% of those surveyed indicated that they were using a new hypervisor in their cloud deployment, compared to the 47% using their existing technology. 

In Gillen's words, this "opens the door for non-installed alternatives such as KVM into VMware-dominated shops." 

This is particularly true for new applications that are born in the cloud, especially public clouds, rather than old workloads being migrated there. We're already seeing a class of applications skip the private cloud altogether, starting up on public clouds like Amazon. And while many enterprises still haven't dipped into the cloud, it's interesting to see what little variance there is between private and public cloud adoption:

Add to this Rightscale's finding that 77% of enterprises are using multiple clouds, and it seems doubtful that any vendor will be able to gently lead enterprises from its virtualization technology to its cloud. Fragmentation is the norm.

The Cloud? It's Complicated

VMware isn't going away anytime soon, in part because the enterprise moves slowly, and in part because VMware has a compelling cloud story for enterprises when they do decide to graduate from simple virtualization to private and public clouds.

But that "graduation" path is messy, with plenty of room for enterprises to find their way to different hypervisors and competing clouds. For these reasons, the virtualization and cloud markets may well be among the most competitive technology markets we've seen in a long time.

Image courtesy of Shutterstock.

As much as we don't like markets being dominated by a single vendor, it's almost as bad to try to choose between a chaotic mix of vendors. That's the current state of the cloud market, and it's giving some prospective buyers fits. For public cloud, Amazon is the early leader, but within the enterprise...? It's not so clear.

The major cloud vendors admit as much. In a recent Quora thread, executives from Eucalyptus, VMware and more debate who leads the Infrastructure-as-a-Service (IaaS) cloud market. Answer? 

It depends.

AWS vs. VMware vs. OpenStack?

Eucalyptus CEO Marten Mickos starts it off, answering the question of "In the IaaS cloud market, who will win between AWS, VMware and OpenStack?" with:

All three camps have their respective strengths. VMware is the undisputed leader in virtualization and more broadly in on-premise infrastructure software. So far they have little to show when it comes to public clouds. OpenStack has huge popularity and the backing of legacy IT vendors. They are fighting the public cloud and the private cloud battle at the same time. Amazon Web Services are overwhelming leaders in public cloud, an industry that is growing fast. AWS hasn’t done much with large enterprises or on-premise environments. They do have Virtual Private Cloud (VPC), Direct Connect (DC), and the partnership with Eucalyptus.

But it's not as if these vendors are operating in a vacuum. As VMware cloud executive Matthew Lodge notes, both AWS and VMware bring existing fan bases (he calls them "power bases") to the cloud party. VMware has an enviable foothold with enterprise IT and AWS owns developers. OpenStack, as Lodge points out, has "undoubted enthusiasm around the project from vendors and early users" but "no strong power base" and still lacks many public clouds built on its technology, and has been particularly slow to gain traction in Europe. 

Cloud Apples vs. Cloud Oranges

Which, of course, is a reminder that at times we're comparing cloud apples and cloud oranges here.

IaaS architect Jason Heiss hones in on this, posing a mock rhetorical question - "In the housing market, who will win between Century 21, Home Depot and a lumber mill?" - and then stressing that each of these cloud providers is "selling different things to different people," concluding "They'll likely all 'win,' in the sense that cloud adoption is still nascent in many companies."

I'm not sure that I agree that all will win, even in Heiss' sense. While the cloud a growing market with lots of room for "winners," enterprises are going to settle on a few vendors, not many. VMware has the "power base" with enterprises, and AWS has the same with developers. Eucalyptus ties the enterprise into the power of AWS through its API, and OpenStack has a great deal of momentum from vendors who want it to succeed against incumbent power bases.

In other words, this game is nowhere near over, and it may be too soon to pick a winner.

Can Any One Vendor Win The Cloud?

And even when we do pick a winner, are we picking a winner in enterprise cloud deployments, public cloud deployments, hybrid cloud deployments, deployments of public clouds built on one's cloud technology, or something else altogether? Defining the market matters, and at present it's not clear that there's any useful way to describe the overall "cloud market" as a coherent thing that any one vendor could possibly win.

So when you read that OpenStack has upended the private cloud market, or read a blow-by-blow account of who's winning between CloudStack or OpenStack, a healthy dose of skepticism may be in order. Not of the analyses, which are often quite good, but rather of the very idea that any particular vendor could win this amorphous market we call "cloud."

Image courtesy of Shutterstock.

Today the race is on to virtualize all aspects of the data center. Dubbed the software-defined data center (SDDC) or sometimes software defined networking (SDN), SDDC is a market IDC projects will top $3.7 billion by 2016.

It's a hot market, too: just this week, Cisco, IBM, VMware, Red Hat and others have banded together under a Linux Foundation-hosted consortium called OpenDaylight. But while this is a significant step toward virtualizing the networking layer of the data center, it may simply be a prelude to the next phase of virtualization: storage.

VMware led the way in virtualizing servers in the data center, creating enormous value for its shareholders over the last decade. Originally acquired by EMC for $635 million in 2003, VMware is now a standalone company with a market capitalization of more than $30 billion. Last year it acquired a leading SDN startup, Nicira, for nearly $1.3 billion. That move scared a lot of data center vendors – primarily Cisco – who don’t want to see VMware dominate networking virtualization as completely as it came to own server virtualization.

Too often overlooked in all the billions of dollars sloshing around servers and networking competition in SDDC is the laggard, storage. Traditional storage is a $10 billion annual business, but until recently it hasn’t made much headway into virtualization.

That may be about to change. 

To better understand the trends shaping the rise of the software-defined storage play, I sat down recently with Dr. Kieran Harty, CEO of Tintri, makers of storage systems for software defined data centers, and one of a core virtualization pioneer. Harty ran engineering at VMware from 1999 to 2006 and his teams created the software products that virtualized the server side of the SDDC equation.

ReadWrite: Remind us again what VMware was trying to do a dozen years ago when your teams were focused on bringing virtualization to servers.

Harty: The basic problems virtualization solved back then we called server consolidation and over-provisioning. Business wanted to move compute workloads from large, costly, proprietary, single servers (usually Sun servers) running one application, oftentimes at only 10% of capacity, to clusters of cheap, commodity, Linux servers. VMware pioneered a technology called the hypervisor that allowed virtualization to make this possible – on the server.

ReadWrite: Today VMware enjoys roughly 90% market share in server virtualization. The spectacular success of server virtualization begs the big question of what comes next. Can the same benefits of virtualization on servers be applied to the rest of the data center?

Harty: This is what gives rise to the concept of the software-defined data center (SDDC) – a data center with infrastructure that is fundamentally more flexible, automated and cost-effective; infrastructures that understand application workloads and can automatically and efficiently allocate pooled resources to match the application demands. Rather than construct data centers full of over-provisioned and siloed resources, a SDDC would more efficiently utilize and share all aspects of the infrastructure: servers, networking and storage.

While servers, and to a lesser extent networks, have embraced SDDC, storage lags significantly behind and continues to cause a great deal of pain in the data center today. Fortunately, some of the key technologies that brought the sweeping changes to servers and networks are taking shape for storage.

ReadWrite: What kind of changes?

Harty: A quick look at some of the most successful disruptive technologies reveals that many of them “crossed the chasm” with the help of a few common key ingredients: standardization, hardware innovation and abstraction. In the case of server virtualization, the standardization of Intel’s x86 platform and the proliferation of the open source Linux operating system massively disrupted the server market. Armed with a new generation of multi-core processors and VMware’s hypervisor technology, server virtualization conquered the data center.  

Networks followed a similar path starting with TCP/IP standardizing the network protocol. Gigabit Ethernet increased transmission speed by an order of magnitude. OpenFlow, which set the foundation of an open and standards-based software-defined networking, paved the way for the most significant changes in networks in several decades. 

ReadWrite: What kinds of changes in standards, hardware innovation and abstraction are leading to disruption in the storage market?

Harty: For 20 years, little has changed in the world of legacy storage designed for physical environments. As data centers become more virtualized, there is a growing gap due to the complete mismatch between how storage systems were designed and the demands of virtual environments. It’s a bit like people who don’t speak the same languages and have a hard time understanding each other - storage speaks LUNs and volumes; servers speak VMs. 

As a result, they don’t understand each other very well. Storage allocation, management and performance troubleshooting for the virtualized infrastructure are difficult, if not impossible with legacy storage. Companies have tried to work around this obstacle by over-provisioning storage which is very expensive and increases complexity.

ReadWrite: Is there where flash technology enters and disrupts storage? Can we power through these legacy storage challenges with performance improvements that are an order of magnitude over those of traditional spinning disk?

Harty: Storage has always been about performance and data management. Flash removes the performance challenges and levels the competitive playing field for storage vendors. Flash enables very dense storage systems that can host thousands of VMs in just a few rack units of space. But flash by itself – without the intelligence – only gets us so far. 

And while some industry players are attempting to make virtualization products adapt to legacy storage through APIs, or retrofit legacy storage to become virtualization aware, neither goes far enough to bridge the yawning gap between these two mismatched technologies – you can put lipstick on a pig, but it’s still a pig. What is needed to solve this problem is storage that has been completely redefined to operate in the virtual environment and uses the constructs of virtualization. In short, VM-aware storage.

ReadWrite: What do you mean, VM-aware?

Harty: Virtualized environments require storage designed for virtualization. Enterprises expecting to get the full benefit out of the software-defined data center need storage that’s simple and agile to manage, while delivering the performance required by modern applications. They will need storage that understands the IO patterns of virtual environments and that automatically manages quality of service (QoS) for each VM. 

We eliminate an entire layer of unnecessary complexity if we stop talking about LUNs or volumes. 

The broad adoption of virtual machines as the data center lingua franca gives us de facto standardization for software-defined storage. The rapid growth and declining cost of flash technology provides the hardware innovation. 

This leaves us with the one last essential missing piece – the abstraction between storage and VMs, an abstraction that understands VMs while being able to abstract and pool the underlying storage resources and deliver the benefits of simple, high performing and cost effective storage. We call that VM-aware storage.

Image courtesy of Shutterstock.

Amazon Web Services is on fire, and EMC and VMware are feeling the heat. So the established enterprise-computing duo is striking back — by launching Pivotal, a joint venture that aims specifically to dethrone the current king of cloud computing.

Pivotal is led by Paul Maritz, the ex-CEO of VMware and a former senior executive at Microsoft. In leading the charge against AWS, Maritz is diving into a cloud-computing mosh pit that will include other tech heavyweights, such as IBM, Microsoft and Oracle.

(See also: Amazon: Can It Stay King Of Cloud Computing Forever?)

Pivotal heads for battle with parent-company assets — database technologies, data analytics and an application platform — it is combining into services that customers can lease to run their own software in the cloud. EMC owns 69% of Pivotal and VMware the rest. The two owners will have to invest a total of $800 million this year and next in order to kick start Pivotal, which Maritz conservatively estimates will reach $1 billion in revenue in five years from $300 million this year.

Amazon's Lead

Those numbers show how long it will take Pivotal to catch up with AWS. While Amazon won't break out the numbers for its cloud-computing unit, analysts say it is lumped inside the revenue category the online retailer calls "other." In Amazon's fourth quarter earnings released in January, "other" accounted for $769 million in revenue for the quarter and $2.52 billion for the year. That's a respective growth of 68% and 64%, according to the International Business Times. 

(See also: VMWare: "If Amazon Wins, We All Lose")

Nevertheless, the market is still young. Most AWS customers today are startups and small and medium-sized businesses. Amazon is expected to shift focus to large companies soon, heading right into EMC's and VMware's sweet spot. This is making both companies very nervous.

During a partner conference in February, VMware Chief Executive Pat Gelsinger warned that if "a workload goes to Amazon, you lose, and we have lost forever," CRN reported. To avoid that kind of customer drain, Pivotal will provide the public-cloud option for VMware customers using its infrastructure technology for private clouds. Supporting that migration is important to EMC, because it owns 80% of VMware.

Pivotal In The Cloud

On paper, Pivotal will provide an enterprise-class cloud-computing platform and infrastructure. The company includes Greenplum, EMC's Big Data analytics division, and Pivotal Labs, the storage company's application development environment. VMware is contributing cloud-computing platform CloudFoundry, and middleware and tools for building and running data-intensive Java applications.

Maritz will have to build a business on top of all this technology, but EMC's and VMware's commitment to Pivotal shows how they believe customer migration to cloud-computing environments outside their data centers is inevitable. The companies also know that failing to have what customers want would be suicidal.

In 2011, Gelsinger, then president and chief operating officer for EMC, said the company did not intend to become a casualty of any major change in the industry.

"The technology industry is ruthless and relentless," he said during an interview at the VMworld conference. "If you are not in front of those major waves of technological innovation, you will become one of the driftwood on the shores of the industry."

In cloud computing, stopping Amazon is how EMC and VMware plan to reach that shore alive.

Image courtesy of Shutterstock

With IBM tossing its might behind OpenStack, the open source software used to run cloud-computing installations is in a strong position to become the dominant platform in the industry.

OpenStack Rising

IBM announced Monday that it will make OpenStack the foundation of its cloud services and software. In backing the open source project, Big Blue joined other tech heavyweights behind the technology, including Hewlett-Packard, Dell, Cisco, Red Hat and Rackspace.

"IBM is the big fish in the sea and for them to make the level of commitment that they did today is a big deal," said James Staten, analyst for Forrester Research. "That's the kind of heft OpenStack needs."

The announcement is likely to send OpenStack's two main competitors VMware and CloudStack, another open source cloud computing platform, into a battle for second place.

“OpenStack has won the race to become the standard, and it has done it rapidly,” Ann Winblad, a venture capitalist and a managing director of Hummer Winblad Venture Partners, told AllThingsD.

IBM And Open Source

IBM has conducted a long love affair with open source software. In 2000, it backed Linux and a year later committed $1 billion to the development of the operating system. IBM's support helped drive Linux into large organizations and made it a viable competitor against Microsoft as a server platform.

"IBM could have the same impact on OpenStack as it did on the Linux world," Staten said.

IBM recognized years ago that open source code fit its business strategy a lot better than proprietary technology. The company draws most of its $100 billion in annual revenue from providing IT services. By basing a lot of its own technology on the code from various open source projects, as well as industry standards, IBM is able to work its hardware and software into what enterprise types call "heterogeneous computing environments" — the combinations of patched-together technology from a variety of vendors typically found in large companies, the segment of the tech market IBM is strongest.

"IBM has really great credibility in the open source community," Gary Chen, analyst for International Data Corp., said. "They really do understand open source."

IBM's First OpenStack Product

IBM followed its announcement with the introduction of its first OpenStack-based product, SmartCloud Orchestrator. SmartCloud is the brand name for IBM's platform for running cloud installations in customers' or IBM's data centers or in a combination of both. Orchestrator is a service customers use to configure the computing, storage and networking resources for cloud applications.

One unanswered question is how IBM will integrate its current SmartCloud code base with OpenStack. In an interview with NetworkWorld, Robert LeBlanc, a senior vice president of software for IBM, waxed mystical in describing how Big Blue will handle the transition.

"We're on a continual journey," LeBlanc said. "But we think this is a major step in that journey."

Cloud Standards

IBM clearly wants to influence OpenStack's technological direction and efforts to develop industry standards for cloud computing, which is still a relatively immature architecture. IBM has formed a 400-member Cloud Standards Customer Council to help push other tech vendors in a direction favorable to IBM. The company says it has more than 5,000 customers running private clouds on its platform.

IBM is also a major player in standards bodies, such as the World Wide Web Consortium and the Organization for the Advancement of Structured Information Standards (OASIS).

While standards are key to making different technologies work together, they won't help companies make the cultural changes necessary to adopt cloud computing and make it work. Delivering applications as a Web service dramatically changes the role of IT departments and affects how employees interact with software, too.

Because of its success in professional services, IBM is in a strong position to help companies make those cultural changes, but it won't be easy. "A lot of enterprises are not ready to hear it," Staten said.

Nevertheless, the momentum in the tech industry is behind cloud computing. The public cloud service market alone is expected to grow 18.5% this year to $131 billion worldwide.

With that much money on the table, IBM plans to become a major player in the market and is betting that OpenStack can help it achieve that goal.

Image courtesy of ShutterStock

Someone has a problem, and it doesn't appear to be Amazon. In a somewhat shocking declaration, VMware CEO Pat Gelsinger told a group of VMware partners that if "a workload goes to Amazon, you lose, and we have lost forever," as reported by CRN. This is true so far as it goes: the more enterprises move applications to the public cloud, the less need they have for VMware's technology, or for other datacenter-bound infrastructure.

But where Gelsinger really tips his hand is addressing why he wants to keep customers out of the public cloud:

We want to own corporate workload. We all lose if they end up in these commodity public clouds. We want to extend our franchise from the private cloud into the public cloud and uniquely enable our customers with the benefits of both. Own the corporate workload now and forever.

To save customers money? To boost their productivity? To benefit the customer in any way? No, no and no.

Instead, VMware's plea essentially translates to "you have to help us lock customers into our platform," as Benchmark general partner Bill Gurley suggests. It's fine for VMware to say such things in the privacy of its boardroom, but on stage? In front of hundreds of partners and the press? Not wise.

Those familiar with the early days of open source will appreciate the similarities to Microsoft's attacks on open source. When Microsoft wrung its hands about the GPL being "bad for the world," it wasn't really worried about customers. It was worried about its business model. Microsoft knew how to compete with cheap. But free? That was difficult.

Now, VMware is up against a highly disruptive competitor, and by its own admission, it's not winning. VMware President and COO Carl Eschenbach said as much as he tried to rally VMware's partner troops: "I look at this audience, and I look at VMware and the brand reputation we have in the enterprise, and I find it really hard to believe that we cannot collectively beat a company that sells books."

And yet, it hasn't been beating Amazon. Not in cloud workloads, anyway, given Amazon may record as much as $3.8 billion in AWS revenue this year, according to Macquarie Capital. Yes, much of Amazon's volume thus far has come from test/development workloads, but that is almost certainly just a starting point. Remember when Linux was only used for edge-of-the-network, non-critical workloads? That didn't last long...

With any technology disruption, there will be winners and losers, but VMware might want to take a page out of IBM's book, which has somehow managed to weather and even thrive despite serious threats to its legacy businesses. Ironically, even old-school SAP might be able to show VMware the way. SAP now lets customers rent its in-memory Hana database as a service on Amazon.

Perhaps Gelsinger could take a page from his own company's PaaS offering, Cloud Foundry. Cloud Foundry, now part of the Pivotal Initiative, is emphatically open source and explicitly eschews infrastructure moorings. Cloud Foundry, in other words, is the antithesis of the lock-in Gelsinger appears to be advocating. "Lock-in to Amazon is bad," goes the reasoning, "but lock in to VMware? More, please."

Not that VMware is the only conflicted company on earth. Nor is VMware doomed. Not by any stretch. The company continues to have a firm hold on the enterprise CIO. But to keep that hold, it might want to be a bit less blunt about wanting to turn that hold into a stranglehold. After all, one big reason for enterprise adoption of first open source, and now the cloud, is precisely the flexibility to get things done without being locked into any company whose guiding principle is to "own the corporate workload now and forever."

Having recently spun out its developer-focused technology into Pivotal Labs, VMware is back at the DevOps table with a $30 million investment in Puppet Labs, a leading open-source vendor of cloud automation tools.  The investment also comes with a partnership whereby the two companies will "work together towards more extensive automation and orchestration across infrastructure and application elements for VMware-based private and public clouds, physical infrastructures, OpenStack and Amazon Web Services." But given that VMware's $30 million investment represents two-thirds of Puppet Labs' total capital raised, the question must be asked: does it also come with serious strings attached?

Knowing Puppet Labs founder and CEO Luke Kanies, probably not.

No One's Puppet

It's not surprising that VMware would want to invest in Puppet Labs. After all, Puppet has become an enterprise standard for automating IT infrastructure. There are scads of big companies using Puppet, including Google, Harvard, and 24/7 Real Media. While the IT automation is increasingly crowded, with Opscode and SaltStack competing for open-source eyeballs and big vendors like IBM and BMC also in the ring, Puppet Labs has become the go-to technology today for companies with serious infrastructure automation needs. With reference to Opscode, specifically, I've heard it said that companies with existing infrastructure use Puppet, whereas startups born in the cloud tend to gravitate toward Chef (Opscode).

All of which fits well into VMware's customer profile. But does VMware fit into Puppet Labs' plans going forward? And how does Puppet Labs retain its independence given that so much of its funding now comes from one vendor, adding to the funding it already raised from Cisco and Google, as well as a smaller amount from VMware in an earlier round?

For those who know Kanies, this is a bit of a silly question.

I've been friends for years with Kanies, and advised the company early on before it had taken funding. Back then it was just two guys (Kanies and co-founder Andrew Shafer, who has since gone on to work at Cloudscaling and OpenStack and is currently working on a stealth startup), and I was fortunate to spend time with them, working through likely business models. Kanies was somewhat reluctant to take funding then, as he wanted his independence. If you know Kanies, you know that independence is a big deal with him. Raised on a hippie commune in Tennessee (yes, really), Kanies isn't the sort of person to sell his soul, or independence, for $30 million.

Not that doing so for VMware dollars would be a terrible outcome. It seems likely that $30M was second prize in VMware's mind; that it would have preferred an outright acquisition. But with the investment, which Kanies says leaves VMware as a minority shareholder, Kanies gets the money to scale Puppet Labs without the strings attached (sorry, couldn't resist). As Shafer related to me over IM:

The cloud transition is creating growth opportunities for next generation systems management products. The alliance will be great for Puppet Labs and great for VMware. It's important to remember that VMware is building a portfolio of core infrastructure pieces that are rooted in open source, and has given them some autonomy even after acquisition. Puppet Labs has a preexistent relationship with a number of these open-source communities, like OpenStack, which should be complemented by this investment.

Or as Kanies tells me, when asked whether this investment means he's locked into VMware-only infrastructure: "Puppet's strength is in supporting heterogeneous environments, whether physical or virtual servers running on-premise or in the cloud, we support a broad array of infrastructure and are agnostic as to whether it comes from Red Hat, Microsoft, or VMware."

Translation?  Kanies is no tool.  Or a puppet.