But this isn't necessarily a bad thing. Using Hadoop for cheap and efficient storage is a perfect starting point for the next phase of Hadoop's evolution. With this summer's expected debut of Hadoop 2.0, changes are afoot that will make information found within data warehouses and unstructured "data lakes" more accessible than ever.

Hadoop As A Big Bucket

Hadoop has been a great system for storing data since its initial adoption as a Big Data tool, but the data-processing framework MapReduce, which requires the creation of Java apps to reach into stored data and pull out the information required, has a high learning curve.

(See also Hadoop: What It Is And How It Works.)

There are other ways to get information out of Hadoop, of course. The HBase database is included in Hadoop, letting users treat data with a database paradigm. And the Hive data warehouse system enables you to build queries in the SQL-like HiveQL query language that can be converted to MapReduce jobs. But Hadoop is still limited by the fact that everything you do in it still has to be done one thing at a time. MapReduce jobs, Hive queries, HBase operations… they all have to take turns.

This is why a lot of vendors tend to frame Hadoop as the bucket in which data is stored, and cast their products as the magical tool to pull out or analyze that data. In fact, while the data bucket metaphor is apt, it has been super-sized among Hadoop users to become known as data lakes or even data oceans. Given the perceived limitations of Hadoop in its present state, it's not a hard sell to make.

But as the Hadoop development community starts ramping up for the next iteration of Hadoop, those limitations are about to be greatly reduced.

Knitting A YARN Solution

For Arun Murthy, the release manager for Hadoop 2.0, the most important change will be upgrading the MapReduce framework to Apache YARN, which will expand what software can be used in Hadoop and how much. Murthy, who is also YARN project lead and co-founder of Hortonworks, explained that "In Hadoop 1.0, everything was batch-oriented. In 2.0, you will now have multiple apps hitting the data inside all at once."

What YARN does, essentially, is divide the functionality of MapReduce even further, breaking the two major responsibilities of the MapReduce JobTracker component - resource management and job scheduling/monitoring - into separate daemons: a global ResourceManager and per-application ApplicationMaster.

Splitting up these functions provides a more powerful way to manage a Hadoop cluster's resources than the current MapReduce systems can handle. It manages resources similar to the way an operating system handles jobs, which means no more one-at-a-time limitations.

With YARN, developers will be able to build apps directly within Hadoop, instead of bolting them on from the outside, as many third-party vendor tools have to do now.

Murthy reported that the Apache Hadoop community is already seeing keen interest from vendors who want to build their apps within the YARN framework that will live directly inside Hadoop and have resources managed by YARN.

Because the Apache Hadoop community is driving the development of the new version of Hadoop, there is no set timeline for Hadoop's progress this summer. Murthy predicted that a "strong beta" of Hadoop 2.0 might be available in June or July timeframe, with a final release perhaps ready by August.

If YARN lives up to its promises, a lot of data lakes and oceans will suddenly be more accessible within the native Hadoop platform, which will greatly streamline and speed up the task of finding useful information. Big Data is about to get even more useful.

You can't have a conversation about Big Data for very long without running into the elephant in the room: Hadoop. This open source software platform managed by the Apache Software Foundation has proven to be very helpful in storing and managing vast amounts of data cheaply and efficiently.

But what exactly is Hadoop, and what makes it so special? Basically, it's a way of storing enormous data sets across distributed clusters of servers and then running "distributed" analysis applications in each cluster.

It's designed to be robust, in that your Big Data applications will continue to run even when individual servers — or clusters — fail. And it's also designed to be efficient, because it doesn't require your applications to shuttle huge volumes of data across your network.

Here's how Apache formally describes it:

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly available service on top of a cluster of computers, each of which may be prone to failures.

Look deeper, though, and there's even more magic at work. Hadoop is almost completely modular, which means that you can swap out almost any of its components for a different software tool. That makes the architecture incredibly flexible, as well as robust and efficient.

Hadoop Distributed Filesystem (HDFS)

If you remember nothing else about Hadoop, keep this in mind: It has two main parts - a data processing framework and a distributed filesystem for data storage. There's more to it than that, of course, but those two components really make things go.

The distributed filesystem is that far-flung array of storage clusters noted above - i.e., the Hadoop component that holds the actual data. By default, Hadoop uses the cleverly named Hadoop Distributed File System (HDFS), although it can use other file systems as well.

HDFS is like the bucket of the Hadoop system: You dump in your data and it sits there all nice and cozy until you want to do something with it, whether that's running an analysis on it within Hadoop or capturing and exporting a set of data to another tool and performing the analysis there.

Data Processing Framework & MapReduce

The data processing framework is the tool used to work with the data itself. By default, this is the Java-based system known as MapReduce. You  hear more about MapReduce than the HDFS side of Hadoop for two reasons:

1. It's the tool that actually gets data processed.
2. It tends to drive people slightly crazy when they work with it.

In a "normal" relational database, data is found and analyzed using queries, based on the industry-standard Structured Query Language (SQL). Non-relational databases use queries, too; they're just not constrained to use only SQL, but can use other query languages to pull information out of data stores. Hence, the term NoSQL.

But Hadoop is not really a database: It stores data and you can pull data out of it, but there are no queries involved - SQL or otherwise. Hadoop is more of a data warehousing system - so it needs a system like MapReduce to actually process the data.

MapReduce runs as a series of jobs, with each job essentially a separate Java application that goes out into the data and starts pulling out information as needed. Using MapReduce instead of a query gives data seekers a lot of power and flexibility, but also adds a lot of complexity.

There are tools to make this easier: Hadoop includes Hive, another Apache application that helps convert query language into MapReduce jobs, for instance. But MapReduce's complexity and its limitation to one-job-at-a-time batch processing tends to result in Hadoop getting used more often as a data warehousing than as a data analysis tool.

(See also Hadoop Adoption Accelerates, But Not For Data Analytics.)

Scattered Across The Cluster

There is another element of Hadoop that makes it unique: All of the functions described act as distributed systems, not the more typical centralized systems seen in traditional databases.

In a database that uses multiple machines, the work tends to be divided out: all of the data sits on one or more machines, and all of the data processing software is housed on another server (or set of servers).

On a Hadoop cluster, the data within HDFS and the MapReduce system are housed on every machine in the cluster. This has two benefits: it adds redundancy to the system in case one machine in the cluster goes down, and it brings the data processing software into the same machines where data is stored, which speeds information retrieval. 

Like we said: Robust and efficient.

When a request for information comes in, MapReduce uses two components, a JobTracker that sits on the Hadoop master node, and TaskTrackers that sit out on each node within the Hadoop network.

The process is fairly linear: The Map part is accomplished by the JobTracker dividing computing jobs up into defined pieces and shifting those jobs out to the TaskTrackers on the machines out on the cluster where the needed data is stored. Once the job is run, the correct subset of data is Reduced back to the central node of the Hadoop cluster, combined with all the other datasets found on all of the cluster's machines.

HDFS is distributed in a similar fashion. A single NameNode tracks where data is housed in the cluster of servers, known as DataNodes. Data is stored in data blocks on the DataNodes. HDFS replicates those data blocks, usually 128MB in size, and distributes them so they are replicated within multiple nodes across the cluster.

This distribution style gives Hadoop another big advantage: Since data and processing live on the same servers in the cluster, every time you add a new machine to the cluster, your system gains the space of the hard drive and the power of the new processor.

Kit Your Hadoop

As mentioned earlier, users of Hadoop don't have to stick with just HDFS or MapReduce. For its Elastic Compute Cloud solutions, Amazon Web Services has adapted its own S3 filesystem for Hadoop. DataStax' Brisk is a Hadoop distribution that replaces HDFS with Apache Cassandra's CassandraFS.

To get around MapReduce's first-in-first-out limitations, the Cascading framework gives developers an easier tool in which to run jobs and more flexibility to schedule jobs.

Hadoop is not always a complete, out-of-the-box solution for every Big Data task. MapReduce, as noted, is enough of a pressure point that many Hadoop users prefer to use the framework only for its capability to store lots of data fast and cheap.

But Hadoop is still the best, most widely used system for managing large amounts of data quickly when you don't have the time or the money to store it in a relational database. That's why Hadoop is likely to remain the elephant in the Big Data room for some time to come.

(See also ReadWrite's Hadoop coverage.)

The Hadoop market is on a tear, growing at a compound annual growth rate of roughly 60%, according to IDC. But why it's growing, or rather, how it's being used, might surprise you. Given all the media hype around Hadoop and its power to predict everything from the optimal number of raisins in your cereal (23) to the exact date of Armageddon (next Tuesday - call in sick), it's perhaps surprising to learn that comparatively few organizations use Hadoop for analytics. Today most enterprises use Hadoop for the pedestrian uses of storage and ETL (Extract, Transform, Load).

Eventually enterprises get to sexy analytics. But we're not there yet. Not by a long shot.

'Poor Man's ETL', 'Unsupervised Digital Landfill', Or Both?

While commonly billed as an analytics tool, Hadoop remains "a poor man's ETL" for the vast majority of enterprises. Yes, there are enterprises running interesting analytical workloads on Hadoop, but these are the exception, not the rule. Hence, while Cloudera cites three common use cases for Hadoop (data transformation, archiving, and exploration, I'm hearing from analysts that 75% or more of the actual Hadoop adoption resides in those first two use cases.

Which is not to suggest such adoption is valueless. Quite the contrary.

The Common Adoption Path For Hadoop

As 451 Research analyst Matt Aslett highlighted at Hadoop Summit, there is a natural progression from using Hadoop to store large quantities of data (i.e., Hadoop as an "unsupervised landfill"), to processing and transforming that data and ultimately to analyzing that data. The fact that most enterprises have yet to get to analytics in any meaningful way is simply a description of where we are in the Hadoop market's evolution.

Indeed, Aslett notes that "attempting to fast forward to analytics, missing out on the processing/integration stage, creates silos and will result in disillusionment." 

We're still early in Hadoop's technological and market evolution, in part due to the complexity of the technology, with 26% of even the most sophisticated Hadoop users citing how long it takes to get into production as a gating factor to its widespread use. Gartner reveals even lower rates of adoption of Big Data projects, often involving Hadoop, at a mere 6%, as enterprises try to grapple with both appropriate use cases and understanding the relevant technology.

Start With What You Know

Small wonder, then, that enterprises are starting with known use cases like storage or ETL before proceeding to more ambitious analytics projects, as Christos Kotsakis suggests. We're still getting comfortable with Hadoop. Applying an unfamiliar technology to a familiar problem makes a lot of sense.

Some day, we'll get to the point where mainstream adopters commonly use Hadoop for significant analytics. But we're not there. Not yet. Just give it time.

Image courtesy of Shutterstock.

The Big Data boom has largely been fueled by a simple calculation: Data + Technology = Actionable Insights, Magic Ponies, and Superpowers. The reality, of course, is far more pedestrian, because while Big Data technology has indeed increased our ability to store and process lots of disparate data in real-time, the technology is only as useful the people managing it. As Bill Wise, CEO of Mediaocean, highlights, the costs of getting it wrong increase as our reliance on data grows.

To be clear, we've long been able to query so-called "Big Data." We've had expensive data warehousing and Business Intelligence tools for many years. The great innovation of tools like Hadoop is that they've made such capabilities available as free, open-source tools that run on commodity hardware, essentially paving the way for anyone and everyone to become a data scientist.

Therein lies the problem.

Taking an influential paper on economics and intelligence efforts around the Boston bombing suspects as background, wherein a few missing rows in Excel and a misspelling of Boston Marathon bombing suspect Tamerlan Tsarnaev's name, Wise points out that "data management tools (i.e., the FBI’s systems and Excel) were undone by fairly simple errors," with terrible results. In other words, as much as we may believe Big Data is as simple as "Input data into Hadoop, out come insights!", the reality depends heavily on the people querying that data.

And the bigger the data, the bigger the likelihood we'll read it wrong, as Wise posits:

[M]ore human/data interaction means a lot more room for error (and inefficiency) around increasingly critical data sets - which... can have very serious results... If Big Data can’t fit hand-in-glove with usability and workflow, a lot of the promise of big data will be empty data crunching. That’s not just a problem for getting where we want to be in the evolution of computing. It’s a situation that can lead to bad data management - which translates into bad economics and, sometimes, far worse.

This confirms renowned statistician Nate Silver's arguments that data doesn't speak for itself, but is instead corrupted by our biases. Worse, the bigger the data set, the more noise to sift through: "the noise is increasing faster than the signal. There are so many hypotheses to test, so many data sets to mine - but a relatively constant amount of objective truth."

Often, misunderstanding our data simply means our businesses will run more inefficiently or, at least, no more efficiently than before. But if Wise is correct, getting our data wrong can have disastrous consequences.

Which means, as I've argued before, that we really need to look inside our organizations for "data scientists," because context is critical to effectively querying our data, as well as knowing which data to collect in the first place. It also means, as Kate Crawford argues in Harvard Business Review, "data scientists should take a page from social scientists, who have a long history of asking where the data they're working with comes from, what methods were used to gather and analyze it, and what cognitive biases they might bring to its interpretation." 

In other words, the more data has the potential to impact our organizations, the more humble and circumspect we should become in using it. The consequences of reading our data wrong scale with the volume and velocity of that data.

Image courtesy of Shutterstock.

It's perhaps one of the industry's great ironies that today's hottest enterprise technology is yesterday's leftovers at Google. Hadoop, an open-source implementation of Google's MapReduce technology, is all the rage in the enterprise as a primary tool for tackling Big Data, and probably will remain such for years to come.

But at Google, MapReduce may already be too slow and not nearly scalable enough.

This isn't news. Mike Miller, CEO of Cloudant, made this point in 2012, and Bill McColl, CEO of Cloudscale, made it two years before that. As McColl argued in 2010, "the people who really do have cutting edge performance and scalability requirements today have already moved on from the Hadoop model."

Which is another way of saying Google lives in the future.

I've told the story before about a wealthy friend telling me his money lets him "see into the future a few years" by affording expensive things today that will be cheap for everyone in the future. In a similar fashion, Google, not to mention other web giants like Facebook and Twitter, is building things today, to solve problems of scale and data processing, that will likely be commonplace for mainstream enterprises tomorrow. 

Today Google's data and scale problems are almost magical. Tomorrow they will likely be average.

Which may mean that peering into the future, whether you're an entrepreneur or a venture capitalist, may be as simple as watching Google. While Facebook releases much of its code as open source, the place to gaze into Google's soul is its treasure trove of published research. There you'll find "Efficient spatial sampling of large geographical tables" and more information on "Spanner: Google's Globally-Distributed Database."

You will see, in other words, the future of enterprise computing, otherwise known as Google's leftovers. 

Image courtesy of AHMAD FAIZAL YAHYA / Shutterstock.

Apple may be that kid who never learned to share, but Microsoft over the years hasn't been much better. While the company has long had a healthy partner ecosystem, if you really wanted tight integration with one of Microsoft's products, you pretty much had to work in Redmond. Microsoft Office worked seamlessly with Microsoft Windows worked seamlessly with Microsoft SQL Server worked seamlessly with Microsoft Sharepoint worked seamlessly with... you get the picture.

Of late, however, Microsoft's underdog status in key markets has made it more amenable to a truly open partner ecosystem, perhaps best exemplified by its open arms to open source.

Nowhere is this more evident than with Windows Azure.

While the old world of Azure looked much like the image above, with Microsoft technology as far as the eye could see, the new Azure looks much different. For one thing, much of the best technology being served up on Azure wasn't written by Microsoft. Really! I'm not joking.

For example, in partnership with Hortonworks, Microsoft has released its first public preview of Windows Azure HDInsight Service, Microsoft's cloud-based distribution of Hadoop, the popular open-source Big Data processing tool. Another example of Microsoft's classic embrace and extend strategy? Nope. This time around, Microsoft promises that HDInsight will be "100%  Apache Hadoop compatible now and in the future."

But Hadoop isn't the only open-source technology included by the Azure team.

More than Just Hadoop

In the olden days, Microsoft would have put all its engineering into supporting its own technologies on a first-class basis. Others might try to catch the Microsoft train, but they'd reverse engineer their way onto the back of the caboose, with just a slight API tweak away from incompatibility. Now it's Microsoft Azure that is adding support for Android, not to mention PhoneGap. All of which follows the Azure team's long-time support for Drupal, various open-source databases, Linux virtual machines, and a range of other open-source software.

"Of course Microsoft supports open-source software on Azure because it's a platform," you argue, "and so Microsoft must support third-party technology as a platform provider."

But that "of course" was lost on Microsoft for years. Through a personal agreement between Bill Gates and Steve Jobs, Microsoft Office came to Mac OS X, but it still hasn't touched Linux. Same with SQL Server. You can get the popular database to run on Linux, but not as a first-class citizen. That's reserved for Windows. 

Windows Azure's Open Community

Beyond directly supporting open-source software on Azure, Microsoft has also opened up its Windows Azure Community Portal to make it easy for partners to add third-party services to Azure, both open and closed. This is a big deal for SMBs and departments within enterprises that have traditionally been Microsoft's mainstay, as BitNami founder and CTO Daniel Lopez told me:

"For customers who are looking to the cloud to run department or workgroup level apps... and who are already customers of Microsoft, the transition to Azure may be simpler and more cost-effective than moving to Amazon. 

"Microsoft has traditionally dominated the SMB market. As SMBs move to the cloud, SaaS cannot meet their customization needs. They need to run their own apps - they just don't want the hassle of running their own servers. Nobody has figured out the 'Application layer' in the cloud yet, but Microsoft is actually in a better starting position than its competitors (Amazon, Google) because it already has a huge installed based and an ecosystem of partners." 

Microsoft, in other words, finally groks "open." In part Microsoft shows this by embracing leading open-source technology like Hadoop or Android, but it's just as clear by its willingness to let partners embrace and extend Azure with other offerings. Yes, Microsoft has long done this with Windows, but it was never a level playing field for some kinds of technology, like open source.

Which is not to say Microsoft has won the public cloud. Today that distinction clearly goes to Amazon Web Services. But while AWS is sexy with the Silicon Valley set, the horde of SMBs and enterprises that have traditionally gone with Microsoft will be looking closely at Azure. Microsoft remains the CIO's top vendor, according to a Piper Jaffray survey. By embracing open source, it stands a chance of being the enterprise developer's top vendor, too. 

The Hadoop market is getting interesting. Last year it was a death match between startups vying to own the heart of the project. Today it's a veritable smorgasbord of big-brand vendors getting involved to ensure they claim a big piece of the Big Data pie. Unlike American youth athletics, not everyone will get to take home a trophy.

Hadoop plays a key role in the burgeoning Big Data market, and represents a $13 billion market by 2017, according to Markets and Markets. (IDC pegs the market much, much lower at $812.8 million in 2016, but its numbers don't seem credible to me as they don't even seem to include Cloudera's sales.) Given that Big Data is hot, and Hadoop's data processing engine sits at its core, there's going to be a lot of money trading hands for Hadoop-related products and services.

Not everyone is going to collect.

SiliconAngle's John Furrier has challenged me on this, arguing that Hadoop is "not a winner take all market." While I, too, can see multiple winners in Hadoop, just as there have been in Linux (e.g., Red Hat dominates license/services revenue, but IBM, HP, and others make arguably more with related hardware, complementary software products, and professional services), markets don't tend toward entropy. They trend toward consolidation.

Today, the Hadoop ecosystem increasingly represents entropy:

• Cloudera, Hortonworks, and MapR remain the early favorites, but with very different approaches. Hortonworks positions itself as the 100% open source player; Cloudera somewhat does the same, but adds in complementary, proprietary bits, mostly around managing Hadoop, to add value to Hadoop (and its top line revenue); and MapR provides a hybrid open source/proprietary Hadoop distribution that swaps out HDFS for its proprietary NFS storage layer.
• EMC Greenplum has been involved with Hadoop for several years, and is set to release a new distribution of Hadoop called Pivotal HD. I've labeled Pivotal HD proprietary, but EMC's Hadoop team has taken issue with this characterization, arguing that PivotalHD is 100% open source, with complementary functionality (like HAWQ) available as add-ons. Point well taken, and I apologize for my misunderstanding. I was wrong, perhaps not surprisingly getting confused by Pivotal HD's product page, which says little about open source. But what seems clear is that customers won't be confused by EMC's value proposition: Hadoop with an advanced SQL query engine to make it easier and more powerful to use.
• Intel just got into the game with its own Hadoop distribution. Basically, you can think of it as Hadoop on (Intel Xeon™ processor, Intel SSD, and Intel 10GbE networking.hardware) steroids.
• For those who don't want to run Hadoop within the datacenter, Amazon offers Amazon Elastic MapReduce (EMR). As of April 2012, EMR was powering over 1 million Hadoop clusters. Presumably this number is much bigger today.
• Many, many others including IBM BigInsights, a range of startups, and more.

Will all of these companies make serious bank on Hadoop? No. Will some of them? Sure.

Ultimately, the winners in Hadoop will be those that invest most heavily in its success, as they will be perceived as the companies best positioned to help would-be customers succeed with Hadoop's complexities. But how they invest is up for discussion. Code to Apache Hadoop? Value-adding extensions?

Success isn't about open source purity, as Gartner's Merv Adrian posits: it's about making customers successful. As we saw with Linux, where Red Hat is both the top contributor to the Linux kernel and the company that harvests the most revenue from distributing Linux, contributing code is a great way to signal to the market that you're a leader and capable of getting code fixes to support customers. Code matters.

But code contributions are not the only way to demonstrate leadership and attract customers. Ultimately, companies that make it easier to get value from Hadoop will win big. There may be more than one such company. Indeed, there almost certainly will be. 

But there won't be 20 of them. Or even 10. Enterprise IT is simply not going to be able to manage a polyglot Hadoop distribution ecosystem. That's not the way markets work. No one wants to be "long tail" vendor, and customers don't want to buy from them, either, as Hugh MacLeod humorously points out on Gaping Void:

Source: GapingVoidArt. Used with permission.

The Hadoop market over the next year is going to be hugely interesting. And bloody.

Image courtesy of Ehab Othman / Shutterstock.

Hadoop, nearly synonymous with Big Data, has many failings. But open source is not one of them. In fact, Hadoop's open-source license remains one of its biggest draws, giving enterprises plenty of reasons to persevere in using it despite its shortcomings. It's therefore hard to see how EMC's new Pivotal HD, essentially a proprietary distribution of Hadoop, can hope to succeed.

Not that everyone agrees with this statement.

Dan Woods, CTO and editor of CITO Research and a contributor to Forbes, argues that embedding Hadoop into EMC Greenplum's massively parallel processing (MPP) database (HAWQ) offers CIOs and CTOs the simplicity they need to be successful with Hadoop. He has a point: Hadoop is complex and somewhat hard to use, which is why Cloudera CEO Mike Olson has argued that most of the world will experience the power of Hadoop through applications, nearly all of which will be proprietary, I might add.

But Olson's argument differs from Woods' argument in at at least one major way: Pivotal HD is enterprise infrastructure, not an application, and enterprise infrastructure is increasingly open source.

There are plenty of reasons for this, but RethinkDB's Alex Popescu nails one critical factor:

Hadoop is so successful despite its complexity [because i]t allows experimenting and trying out new ideas, while continuing to accumulate and storing your data. It removes the pressure from the developers. That’s agility. It’s highly appreciated.

In other words, a big reason for Hadoop's success is its open-source license, which permits a hefty amount of experimentation without having to get an enterprise license from EMC, Oracle, or any of the other incumbent infrastructure vendors.  

EMC's Scott Yara tries to deflect criticism of its proprietary foray into Hadoop by declaring "We're all in on Hadoop, period," but as 451 Research analyst Matt Aslett counters, "I have no doubt that EMC Greenplum is 'all in' on Pivotal HD, but that’s not the same thing at all."

Take this away by building a proprietary Hadoop distribution, and EMC has basically erased the very thing that made Hadoop workloads proliferate in the first place. EMC also cuts itself out of the standard adoption cycle for Hadoop, as Redmonk analyst Stephen O'Grady suggests, "Certainly there will be customers whose needs will dictate the adoption of a unique solution like Pivotal HD, but how many will that be relative to the segment whose adoption cycle begins with the download of one of the free Hadoop distributions?"

Today, Hadoop is one of the industry's hottest job trends. Even in absolute job numbers, it's about to pass EMC-related job posts:

Hadoop,emc Job Trends

Hadoop jobs - EMC jobs

Enterprises aren't hiring for EMC's brand of Hadoop. They're hiring for the open source Hadoop. This matters.

Perhaps EMC feels that Hadoop's brand is big enough now that enterprises essentially understand it and are ready to move on from experimentation to full-scale adoption. In this EMC is likely to be disappointed. According to recent IBM survey data, only 6% of enterprises have two or more Big Data projects underway (likely, though not explicitly, involving Hadoop in some way), and a mere 22% are running pilots to test the efficacy of their Big Data strategies. Everyone else is in full-on planning mode.

By creating a proprietary Hadoop distribution, EMC just dramatically limited its access to the 94% that are still in Big Data education and trial mode. Yes, it has a gargantuan sales force. No, they're simply not going to be able to reach would-be customers as efficiently as an open-source distribution model does.

But maybe EMC hasn't gone proprietary to more effectively monetize Hadoop interest, and instead sincerely believes, like Woods ("open source development has its limits"), that complex infrastructure problems are a poor match for open source. History has not been kind to such thinking, as Aslett sarcastically implies:

"Enterprise products" always prevail over open source. onforb.es/VgQ0Cq That's why Linux has been such an abject failure versus Unix.

— Matt Aslett (@maslett) March 1, 2013

EMC has seemingly bottomless resources to throw at Hadoop, and every incentive to do so. It's a smart, highly successful company and no doubt will prove successful with Pivotal HD. However, I can't see it ever dominating an open-source infrastructure market with a proprietary distribution. Open source is the foundation for today's most interesting markets, from Big Data to mobile to cloud computing. It's unlikely that EMC will somehow stem this tide with a proprietary product, no matter its short-term performance or functionality advantages.

There's no beating around this bush. Today Hortonworks announced a new beta version of its Hadoop Data Platform that will run on Microsoft Windows Server, a move that shows Microsoft's own Big Data efforts will forever be connected to open source innovation. This is a highly significant – even expected – move in the big data sector, but also a very strange one.

Hadoop, of course, is an open-source software architecture that supports distributed computation jobs on huge data sets – in other words, classic Big Data work. Hortonworks, meanwhile, is one of the bigger Hadoop vendors in the market, even if that's more in terms of innovation than sales, where it trails Cloudera. Hortonworks founder and architect Arun Murthy is one of the original Hadoop coders who came out of Yahoo back in the day, and he also serves as the VP of the open source Apache Hadoop project at the Apache Software Foundation.

Which all means that any major platform move like this is sure to impact the rest of Hadoop development and, by extension, the rapidly growing Hadoop ecosystem that's driving much of the big data sector.

Why Windows?

Until today's announcement, Hadoop of any flavor typically ran on a Linux-based machine (physical or virtual). This made a lot of sense, since one of the big advantages of Hadoop is the capability to expand its data warehousing over any number of clustered computers. When those clustered machines are running Linux, it's all but frictionless to add more, both in in terms of licensing cost (which is free) and configuration (which is easy).

But when the underlying operating system is Windows Server, licensing – i.e., explicitly not free – would seem likely to create a lot more friction when someone tries to build a Hadoop cluster. Wouldn't using Windows Server as the OS for a Hadoop system be too expensive?

David McJannet, VP of marketing at Hortonworks, doesn't seem to think so. McJannet's concern was that too many Windows-based shops out there were shying away from Hadoop because they didn't want to deal with adding Linux clusters and the related hassle of managing them. So assuaging those concerns was one big reason Microsoft has been working with Hortonworks over the past 18 months.

The sheer number of Windows installations was also a major issue. McJannet said that a "majority of servers" were running Windows in the enterprise now. In its press release, Hortonworks cited IDC data thusly: "According to IDC, Windows Server owned 73 percent of the market in 2012 (IDC, Worldwide and Regional Server 2012–2016 Forecast, Doc # 234339, May 2012)."

It is not clear just what server class this 73 percent represents, since the report itself costs $4,500, and is thus a little hard to access. File servers? Application servers? It's sure not web servers, where according to Web analytics from Netcraft, Microsoft currently has 16.93% of the marketshare, dwarfed by Apache's 55.26% marketshare.

McJannet also said Hadoop on Windows would make data exploration easier. Using SQL-based queries that can now directly integrate with the Hadoop Distributed File System (HDFS), products like SQL Server and Excel can tap straight into Hadoop-stored data, enabling end-users to more easily navigate vast stores of data in Hadoop clusters.

Embracing Open Source

This is not Hortonworks' first foray into Windows land. Late last year, it released the Windows Azure HDInsight product – essentially Hadoop for the Azure cloud platform.

As odd as it may seem to see Hadoop on Windows Server, the move makes a lot of sense from Microsoft's side. The company has needed a Big Data entry ever since it decided to drop its own Dryad data warehousing framework back in 2011. Some observers have expected this day ever since a year ago, when Microsoft announced it would build in tools within SQL Server to connect to Hadoop.

McJannet emphasized that to date, Microsoft was playing well with others within the open source development model that Hadoop uses, so much of its innovation will cycle back to the rest of the Hadoop community.

If so, you can expect to see more Hadoop vendors to announce their own connections to Windows in the near future.

Image courtesy of Shutterstock

Big Data is, well, big these days. (Maybe too big for its own good.) So it should surprise exactly no one that Red Hat, itself a big open-source enterprise software and services company, wants to wear the Big Data hat, too. Which is exactly what it announced in an hour-long Webcast on Wednesday.

So, terrific. Red Hat is now a Big Data company – at least, that is, if you believe Red Hat. What does that mean, exactly? Surprisingly little.

Let's put it this way: Red Hat's big news was that it is open-sourcing its storage plug-in for Hadoop, a popular open-source software framework that supports both distributed data and distributed applications. (Simpler explanation: Hadoop makes it possible to run data-intensive applications physically near the data itself, which greatly speeds things up because it's no longer necessary to shuttle great piles of data across a network.)

As a step toward transforming Red Hat's own storage file system into a "fully-supported, Hadoop-compatible file system for big data environments," as the company puts it, this is doubtless noteworthy. As something concrete on which enterprise customers can, uh, hang their hat, it leaves a lot to be filled in later.

That wouldn't be a problem if Red Hat had provided other noteworthy details, such as a road map for its development of what it calls an "open hybrid cloud." It's a nifty enough idea, essentially amounting to a data environment that would help business move their applications from in-house servers to those offered by cloud providers such as Amazon's Web Services unit without having to rewrite them.

But while Ranga Rangachari, Red Hat's vice president of storage, talked up the open hybrid cloud and the "robust network" of partners Red Hat plans to work with to make it a reality, he had nothing to say about time frames or even the identities of its partners. "Just stay tuned as we come up with more definite dates and times and start – we'll absolutely make those partners available," Rangachari pleaded at the end of the event.

Maybe Red Hat is saving the details for the next roundup.

(See also Big Data: Red Hat's Jim Whitehurst Looks 20 Years Into The Future)

Image courtesy of Shutterstock.

You do realize that you work for the only company on the planet that isn't leveraging Big Data? That everyone else is gaining competitive advantage by aggregating cash register receipts, the weather in Miami, your sister's Facebook posts and the average shelf-life of Lindt chocolate? That you, in fact, are the world's biggest Big Data failure?

While not generally expressed in this way, in my conversations with IT and line-of-business executives, such sentiment comes out in the subtext of what they say. The media (over)hypes Big Data and IT executives then come to think they must be doing something wrong, as Gartner analyst Svetlana Sicular has found in her conversations with clients. As the current thinking goes, if enterprises don't have warehouses overflowing with data, with data scientists madly crunching the data and coming up with "actionable insights," they're doing it wrong.

One Big Landfill

As just one example, I recently heard one executive at a Fortune 100 company say, "Hadoop is our unsupervised landfill." Spoken like a man who knows his data is important, but isn't quite sure why or how. So his company just stores everything in the hopes that all that data will one day make sense.

This is a reasonable response, given the pressures, but it's actually okay to not have The Big Data Answer. Odds are, your enterprise needs to figure things out over time, even without the mythical (and expensive) data scientists we keep reading about. Sicular argues that "Organizations already have people who know their own data better than mystical data scientists." Give these in-house experts the top tools for Big Data, described in a recent Dice.com job trends report, all of which happen to be open source, and let them iterate toward understanding the data. 

Indeed, open source is the key here, not how big your data is.  

Exploration, Not Exploitation

Alex Popescu nails it when he posits, "Hadoop is so successful despite its complexity [because i]t allows experimenting and trying out new ideas, while continuing to accumulate and storing your data." Unlike with proprietary technology, in open-source Big Data technology you don't have to sign any contracts, fork over any money, or do any of the things typically expected with enterprisee software vendors. You just download and explore.

This fact was underlined for me at a Big Data panel in Chicago this week, which featured Dr. Philip Shelley, CTO at Sears Holdings. Sears is arguably one of the industry's top pioneers when it comes to Big Data, and he insisted that open-source tools like Hadoop were critical to the company iterating its way to Big Data success. Things have gone so well that he has decommissioned millions of dollars in IBM Netezza and other proprietary technology to focus on Hadoop as its data hub. As he said, "We no longer have to budget for capital expenditures" for Big Data initiatives."

That's impressive.

Yes, data volume is growing. But that's cause for exploration and iteration, not frustration and despair, following Sears' example. You're not alone if you don't yet know what to do with all your data, or if you're wondering if you have enough to bother. As Brian Proffitt has pointed out, small companies with less than gargantuan data troves can also benefit from Big Data technologies, because "big" isn't really about size at all. It's also about variety and velocity of data, among other things.

Or, as Edd Dumbhill ably notes, "'Big data' really means 'smart use of data'."

That "smart use" will almost always involve open source, as explained above. But it should also involve the understanding that you're not in a race to amass data and to recruit data scientists to decipher it. Big Data is an iterative process of using (mostly) open-source technologies to store and analyze data in different ways, learning from peers and from your own experience. It needn't be a landfill of buzzwords.

Image courtesy of Shutterstock.

Gartner research director Svetlana Sicular thinks Big Data is about to plummet off the "peak of inflated expectations" into the "trough of disillusionment." Perhaps. But other data from Twitter and job trends suggest a much more complicated picture.

Sicular reaches her conclusions about Big Data based on a series of conversations with IT professionals over the past few weeks, in addition to a roundtable with Hadoop vendors Cloudera, Hortonworks, and MapR. In discussing Hadoop, the vendors suggest that "MapReduce has always been Hadoop’s bottleneck or that Hadoop is 'primitive and old-fashioned,'" apparently disillusioned with the state of Big Data's poster child/elephant. 

This could be chalked up to the Hadoop vendors simply acknowledging that despite being an excellent technology, Hadoop still has a ways to go. But Sicular's conversations with enterprise business analysts are more damaging:

My most advanced... Hadoop clients are also getting disillusioned. They do not realize that they are ahead of others and think that someone else is successful while they are struggling. These organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions... Formulating a right question is always hard, but with big data, it is an order of magnitude harder, because you are blazing the trail (not grazing on the green field).

And yet, these same companies don't seem to be giving up on Big Data. 

For example, DataSift plowed through 2.2 million Twitter mentions by more than 981,000 authors, as Ovum analyst Tony Baer reports, finding that positive mentions of Big Data vendors outnumber negative mentions by 3-to-1. And while Baer acknowledges that "Twitter streams are not a scientific focus group for detecting brand awareness, they provide a valuable window on market thinking." Indeed, given the levels of Big Data hype, it's surprising that the overall mood about Big Data remains overwhelmingly positive.

So much so, in fact, that enterprises are paying a premium to hire job candidates with Big Data-relevant technology skills, as Dice.com's 2012-2013 annual salary survey reveals. Job candidates with Big Data technology expertise command an average salary of $100,000, while other hot technologies like cloud/virtualization ($90,000) and mobile ($80,000) yield lower salaries. As Alice Hill, managing director of Dice.com, asserts, "We’ve heard [Big Data] is a fad, heard it’s hyped and heard it’s fleeting, yet it’s clear that data professionals are in demand and well paid."

While Gartner clearly has a valid point that Big Data's outsized expectations are sure to crash into reality at some point, it's also clear from jobs data, in particular, that enterprises see enough value from their data that they're willing to pay up for expertise that can analyze it. Will they be disappointed? Possibly. But the jobs data indicates we have yet to plummet into Gartner's "trough of disillusionment."

IBM spends over $1.5B every quarter in research and development (R&D) expenses. SAP? Closer to $700M. Oracle? Assuming we don't count the tens of billions it spends buying other companies, its actual quarterly R&D budget comes in just over $1B - around $900M. Microsoft, which has printed billions of dollars in profit each quarter for eons, spends more than them all, topping $10B each year.

And yet not one of these companies is responsible for the biggest advances in enterprise technology in the past decade.  Cloud computing, Big Data, mobile... they're all being invented elsewhere, not by the enterprise behemoths.

Maybe they're doing it wrong?

Lots Of R, Little D

Take the cloud, for example. Microsoft claims to invest 90% of its R&D budget on cloud computing, but it is Amazon, Microsoft's penny-pinching, book-retailing neighbor, that sets the terms for innovation in cloud computing. Amazon launched EC2 back in 2006, when it was spending a measly $132M or so each quarter on R&D. 

Even if we dismiss Amazon, where else are we seeing other cool advances in cloud computing? Netflix, for one, which just released Janitor Monkey to help Amazon Web Services (AWS) users dispose of their unused AWS resources, and the video company previously released Chaos Monkey, which helps enterprises plan for and architect around cloud failure.

Notice the word I used? "Released." It means these tools were open sourced, not put out for sale. That's how innovation seems to happen in the 21st Century.

In large part innovation comes with an open-source license because it's a by-product of businesses that heavily rely on technology, but don't actually sell technology.  It's 'trickle-down innovation' from the web business community.

VMware's Brad Hedlund spotlighted this trend back in 2011, when the enterprise awoke to discover it had problems that the web giants had already solved:

As properties such as Yahoo!, Google, Facebook, Amazon became great successes, their architects and software engineers realized that they had moved mountains...The tremendous problems of efficiently running large scale applications on low cost infrastructure had been solved... At the very same time, enterprise IT begins to encounter some of the very same problems solved by the large web provider, such as scalable data warehousing and analytics (so called “Big Data”). Additionally, the software driven distributed systems that solve problems of infrastructure efficiency and management at very large scale could also be applied to infrastructure at a smaller enterprise IT scale (why not?). And finally, the cost savings of an application infrastructure designed to operate on low cost commodity hardware can be realized at any scale, large web or enterprise IT.

Filling The R&D Gap

Companies like Cloudera, DataStax and others stepped into this gap, taking the open source (or, in the case of some of Google's research, open knowledge) projects from the web and applying them to the enterprise in the form of Hadoop, Storm, NoSQL databases, etc.  All of it developed at a comparative pittance to enterprise incumbents' R&D budgets.  All of it available free of charge on commodity hardware. 

As an industry, we're richer for such open source innovation.  Ironically, so are the enterprise IT vendors, who are investing tens to hundreds of millions of dollars in Hadoop and other open source data and cloud innovations, even as they continue to sink tens of billions of dollars into their homegrown R&D. Maybe it's time for them to reevaluate how they do R&D.  Maybe, like Facebook or Twitter, they should release their R&D on GitHub as open-source code.  At the least they could, like Google, centralize their research on the web, making it easily available to all.

Maybe, just maybe, they'd realize that it doesn't actually matter how much money a company spends on R&D.  What matters is whether it can execute and turn ideas into winning products, as Brad Reed argues, and whether it can help foster community around promising open source efforts.  

This is what new enterprise IT - Facebook, Twitter, Google and Yahoo - demonstrates.  It remains to be seen if IBM, Microsoft and other traditional IT vendors are paying attention.  Unless they do, they'll lose relevance as a new breed of innovative startups emerge to claim the strategic largesse of CIOs' budgets.

With all the recent teeth gnashing about startup investment shifting from consumer to enterprise technology, it's worth noting that successfully cracking the enterprise market is no easy task:

• 70% of the U.S. economy hinges on consumer spending. Even with the pending fiscal cliff, it's kind of hard to ignore the numbers. 
• Enterprise technology is not a short game.

Unlike most consumer technologies, enterprise infrastructure and applications run on a much longer upgrade cycle: 5-7 years. While you might ditch your smartphone every year or two for a newer model, few companies are willing to swap out their CRM systems, storage or security technologies that quickly.

Switching behavior is both the most complicated and important subject in the enterprise technology market. Even if enterprise customers have good reasons to be unhappy with their technology vendors (e.g., lack of innovation, price gouging, poor support), their business runs on that technology. This makes them highly incentivized to see existing vendors address any issues and continue the relationship. As we all know, moving's a bitch.

Of course, enterprise tech is a rich, rewarding game, so it's worth exploring the strategies startups can use to overcome the barriers to switching in the enterprise market:

1. Transformational Technologies. The ultimate startup is the one that changes the game on an incumbent in such a way that the latter neither can block nor retaliate. Classic examples include Virtualization and Software-as-a-Service (SaaS). Because virtualization decouples compute functions from hardware (while running on top of the hardware), it is the ultimate disruptor because it's non-invasive. SaaS eliminates the stickiness of packaged software - and the lucrative support contracts that go along with it. Interestingly, while there tend to be many attackers in Virtualization and SaaS, only a few players tend to win big. Very big: witness VMware and Salesforce.

2. Changing Product Cycles. Catching technology giants in product transition cycles is one of the most effective ways to insert new technologies. However, this usually requires an outside force to speed insertion. Earlier in my career, Intel Centrino drove the need for enterprise Wi-Fi and forced an architectural change. In 2013 you can see many great examples of this idea, including Palo Alto Networks, Splunk, ServiceNow and Workday. These transition cycles don't last forever, though. Over time the incumbents typically build or buy their way into the new product segment and the situation stabilizes until a new cycle begins.

3. Trojan Horses. Sometimes a new enterprise IT category emerges in an indirect way. Cloud infrastructure eliminates the need to buy IT hardware and software; the rental model emerged as form of shadow IT for specific projects that could not wait for corporate IT to respond. It also became the preferred approach for brand new businesses (Netflix streaming). Amazon Web Services and Rackspace, two big early winners in cloud computing, sell computing cycles by the month, payable with with a credit card - often bypassing traditional IT purchasing processes. Once established, Cloud and SaaS vendors can then turn their attention to selling to mainstream IT.

4. New Buying Centers. The multi-hundred billion-dollar enterprise IT game now pivots on competition for the IT "stack," as we shift from the Client-Server/Web mobel to cloud computing. This change has created a new class of IT decision makers such as the "cloud architect." As companies move more to the cloud, this new IT leadership category drives key decisions for enabling new applications, also driving the buying all of the underlying IT components. And these new buyers may not be as wedded to the incumbent suppliers as were the decision makers they supplant.

5. The Consumerization of IT. The iPhone led to a watershed change both in enterprise mobility and computing. Not only did it challenge corporate purchasing patterns ("I buy, you enable," also known as BYOD, or Bring Your Own Device), it eliminated a final barrier to what constituted a business device. This is less about "consumerizing" enterprise IT, but rather, adapting enterprise IT to leverage consumer technologies. In addition to mobile devices, apps are challenging the application market for business software.

6. Coalitions of the Willing. For most small companies, hiring a large enterprise sales force and entering a year-long acquisition cycle is likely to be an expensive exercise in futility. Sure, you might be able to make a living selling to universities, hospitals and niche verticals, but attacking the Fortune 500 requires friends who need another reason to re-engage in a selling conversation. Manufacturing and strategic partnerships with hardware makers made a lot security companies rich during the client-server era (e.g., McAfee, Symantec). Today, companies like Box are changing the game through new kinds of partnership integrations.

Frontal assaults are the hardest attack strategy for an enterprise startup. Attacking a powerful technology company's profit sanctuary tends to piss them off. If you can pull it off, it might just get your company acquired, but run a big risk of perishing in the attempt.

That's why this tends to be the strategy of large companies (e.g., HP's acquisition of 3Com to attack Cisco) and does not have a great track record. The assault on the business PC by iOS and Android tablets and smartphones may turn out be a more successful example, but, Apple and Google and Samsung are hardly startups.

It can be done, of course. Many decades ago, Microsoft's PC operating system was such a technology and for a generation, a small company in Redmond changed the world. (With a big initial boost from IBM, of course.)

Current technologies that might have the power to force enterprises to switch and create hugely successful startups include Apache Hadoop, Network Virtualization, Flash Storage, and Cloud Storage and Collaboration. That's where I'd look for the next big thing.

Image courtesy of Shutterstock.