Up until the point where data starts making sense, by definition, it doesn’t make much sense. Information requires analysis. And despite its name and its mission, the Web presents more unstructured, unrelated data than any information source ever created.
So it shouldn’t surprise anyone that the biggest trend in data management today surrounds analysis of data outside of databases. We heard this two weeks ago from Oracle at its OpenWorld conference, and we’re hearing it again today with Oracle’s acquisition of an analysis tool provider named Endeca.
The key to the acquisition is a hybrid search/analytical engine called MDEX, which underlies a series of Endeca products ranging from general purpose to vertical and task-specific. For example, Endeca produces a data analysis engine for criminal intelligence services.
“The MDEX Engine shares many of the characteristics of a state-of-the-art database, yet is written for the unique constraints of unstructured, structured and semi-structured data,” reads an Endeca white paper on its core technology (PDF available here). “The MDEX Engine is written around a vertical record store to operate at high speeds and at scale. This structure is then stored in columnar form and achieves high compression, and is highly memory efficient for optimizing the delivery of results.”
Comparison of common database management schemes to Endeca’s MDEX engine. [Courtesy Endeca]
If this kind of acquisition sounds, well, strangely familiar… Last August, HP under then-CEO Léo Apotheker was willing to shutter its webOS devices project and redirect its attention toward acquiring Autonomy, a producer of an analytical search engine.
What’s different in Oracle’s case is that MDEX operates a “vertical” data store, “vertical” in the sense of how fields are organized. Rather than Oracle’s classic tables, MDEX stores long trains of data in key/value pairs, which puts MDEX in some respects on a par with scalable databases like Hadoop. With one buyout, Oracle may put itself at or near par with HP on one level, and Microsoft on another – Microsoft having just announced its partnership with Hadoop parent Hortonworks. (Microsoft acquired FAST Technology, an enterprise search tools provider, back in 2008, although so far the most prominent product of that matchup remains a search add-on for SharePoint.)
“The Endeca Latitude MDEX Engine enables users to explore data and content in an unconstrained and impromptu manner and to rapidly address new questions that inevitably follow every new insight,” the newly acquired company states. And Oracle’s position is as follows: “The combination of Oracle Business Intelligence and Endeca Latitude is expected to provide a comprehensive business intelligence foundation and analytic applications, bringing together information from structured and unstructured data sources.”
Ovum technology analyst Mike Davis believes that the exploding quantity of unstructured data, coupled with the rising tide of independently developed analytical and management tools for that data, caught major manufacturers such as Oracle and Microsoft sleeping, or at least snoozing while officially working on their own catch-up products. “We believe it is about time that Oracle ‘bit the bullet’ and gained itself some high-end capabilities, and a referenceable customer base for enterprise search, rather than continuing to develop [its own] Secure Enterprise Search,” Davis said in a statement for RWW this afternoon. “Endeca was the last of the ‘big’ independent enterprise search engines, Autonomy and FAST having already been swallowed by HP and Microsoft, respectively. We believe this continues to illustrate that even the biggest software companies often fail to get their innovation fast enough internally.”
