In A World Of Open Source Big Data, Splunk Should Not Exist

Splunk should not exist. In a world awash with innovative, open-source Big Data technologies, Splunk remains expensive, proprietary ... and hugely popular. Others may zig, Splunk zags. And yet it works. 

Oh, does it work.

Who Knew Log Data Could Be So Valuable?

In 2013 Splunk pulled in roughly $200 million in revenue. While still not profitable—Splunk actually seems to become less profitable each year—the company is sitting on over $300 million in cash and has a stock price that keeps defying gravity.

All for mining log data and making it searchable. 

That, of course, doesn't quite do Splunk justice. As the company says, Splunk "enable[s] organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices." Surprisingly, given how much hype we have to wade through in the Big Data space, Splunk actually does this, which has over 6,400 organizations to pay it serious cash to help mine their machine data.

Whither The Open Source Competition?

But this shouldn't be. Open source owns Big Data, from Hadoop to NoSQL databases. How is it that a proprietary, expensive licensed software business can thrive?

It may be that there simply isn't a good alternative, similar to how VMware was able to dominate virtualization for so long despite the rise of open-source competitors. Enterprises want open, but they want software that "just works" with minimal fuss even more. And they'll pay for it.

When I asked my Big Data-filled Twitter followers, that's what I kept hearing:

So says Vijayasankar, the global vice president of Big Data at SAP, but he's not alone. As Andrew Clay Shafer notes, Splunk is solving a problem that simply hasn't been solved by open-source alternatives:

This jibes well with the analysis of Peter Goldmacher, analyst with Cowen & Co.:

For most customers and prospects, exploring the opportunity to leverage machine data is a new endeavor. There is a significant amount of risk for companies trying to leverage machine data for the first time, and in our conversations with customers, IT departments are unwilling to compound project risk with vendor risk. Splunk is the safe choice. While many Splunk customers continue to express frustration with how expensive the product is, price doesn't seem to be a top purchasing criteria

Expensive but necessary, seems to be the mantra.

Solutions, Not Raw Technology

Not only has Splunk not been commodified, but it's also not quite as closed as it first appears. As Jacob Singh, regional director for Acquia, highlights, Splunk has "great APIs, so the proprietary part is not a huge issue." In other words, by making Splunk services permeable to third-party software, Splunk gets around the need to make its source code available. 

Another reason Splunk succeeds despite open-source Big Data tools like Elasticsearch or even Hadoop is that it offers enterprises a clear reason to use it, as Prasad Athawale notes. As detailed in a recent post, Hadoop's biggest roadblock may well be that people can't figure out what they're supposed to do with it. While this is changing—Cloudera, for example, advocates using Hadoop at the heart of an enterprise data hub—Splunk's value proposition is clear and well-established. 

Splunk won't always have this advantage—search tools like Elasticsearch are maturing quickly. But Splunk still has years before open-source search tools will catch up, according to Linux systems engineer Jeff Schroeder, giving Splunk more time to build out its ecosystem and products.

Sometimes Open Is Not Enough

Which is a great reminder that even despite open source's outsized influence on emerging trends like Big Data, cloud computing and mobile, open is not enough. Especially in new markets, enterprises aren't looking for low-cost commodities. They want high-value solutions to tangible business problems. 

Hence, while Android has boomed, it has come at the price of openness. Google tightly controls its development. And while OpenStack is mounting a serious challenge for cloud computing dollars, today it is Amazon that dominates the market with its promise of fast-paced innovation and low prices.

Does the world want an open-source Splunk? Probably. But will it settle for a Splunk that offers high value, even at a high price? Yes. At least, until the market is ready for commoditization. 

Image courtesy of Shutterstock