The proliferation of non-relational databases in the tech sector these days could lead you to think that these data management tools (also known as NoSQL databases) are eventually going to make traditional relational databases extinct.

Not so. Each of these database types is best suited for very different types of workloads, and that's going to prevent either one from tromping the other into the dust. Which means that IT and other managers are going to have to figure out which approach is best suited for the task at hand.

In this two-part series, I'll examine the capabilities of both NoSQL and relational databases to help you make the right decisions for your organization.

"NoSQL"? More Like "Not Only SQL"

Right off the bat, NoSQL databases are unique because they are usually independent from Structured Query Language (SQL) found in relational databases. Relational databases all use SQL as the domain-specific language for ad hoc queries, while non-relational databases have no such standard query language, so they can use whatever they want. That can, if need be, include SQL.

NoSQL databases are designed to excel in speed and volume. To pull this off, NoSQL software will use techniques that can scare the crap out of relational database users — such as not promising that all data is consistent within a system all of the time.

That's a key result of using relational databases, because when you are conducting a financial transaction, such as buying something on Amazon, databases have to be very sure that one account is debited the same amount that another account is debited at the same time.

Because so much of this back-and-forth read-write activity is needed in a single transaction, a relational database could never keep up with the speed and scaling necessary to make a company like Amazon work.

According to Bob Wiederhold, CEO of NoSQL database vendor Couchbase, the architecture and features of NoSQL databases bring four key advantages to NoSQL users.

Go Big Or Go Home

Easier scalability is the first aspect highlighted by Wiederhold. NoSQL databases like Couchbase and 10Gen's MongoDB, he said, can be scaled up to handle much bigger data volumes with relative ease.

If your company suddenly finds itself deluged by overnight success, for example, with customers coming to your Web site by the droves, a relational database would have to be painstakingly replicated and re-partitioned in order to scale up to meet the new demand.

Wiederhold cited social and mobile gaming vendors as the big example of this kind of situation. An endorsement or a few well-timed tweets could spin up semi-dormant gaming servers and get them to capacity in mere hours. Because of the distributed nature of non-relational databases, to scale NoSQL all you need to do is add machines to the cluster to meet demand.

There's No Need To Fear

Performance is another way that NoSQL databases can excel. First, every time you add a new server to a NoSQL database cluster, there is performance scaling by virtue of the fact that you're throwing another processor into the equation.

Beyond the scaling advantages, the very architecture of NoSQL tools aids performance. If a relational database had tens or even hundreds of thousands of tables, data processing would generate far more locks on that data, and greatly degrade the performance of the database.

Because NoSQL databases have weaker data consistency models, they can trade off consistency for efficiency. In Wiederhold's social gaming example, if a user updated his or her profile, there's no real degradation of game performance if that profile's new info isn't updated across the entire database instantly. This means that resources can be dedicated to other things, like tracking down that orc that's about smack you around in-game.

Objects Of Desire

When applications developers have to work with data in relational databases, it can at times be troublesome due to data mapping and impedance issues. In NoSQL databases, this is not usually a problem, because data is not stored in the same manner.

With document-oriented NoSQL databases, for instance, data is stored in just that format: documents. And since documents are objects, after all, then programmers who tend to think in object-oriented terms are going to be much more familiar with manipulating such data.

That weaker consistency model helps programmers, tool, since their apps don't have to rigidly conform to data consistency requirements. That makes coding much simpler and (by extension) much faster.

Downtime? What Downtime?

Wiederhold's fourth example is a new one to the NoSQL advantage list. It's something that non-relational databases weren't specifically designed to do, but at which they've nonetheless turned out to be proficient.

Because of their distributed nature, Wiederhold says, NoSQL databases can be pretty much always on. This is a huge advantage for web- and mobile-based businesses that can't afford to be down for a single moment.

With some advanced planning, software updates and hardware upgrades can be performed while the database is still running hot. Try doing that with a relational database without taking it down, and you're in for a world of trouble.

get section where name = Conclusion

NoSQL databases' advantages sound great, and they are — for the right use case. If your company has a data set that will remain relatively constant in size, or that only grows slowly, you'll have little need to migrate to a non-relational system.

If sudden growth, agile development or high performance are something your organization will need for its data, then you should take a look at the NoSQL class of databases and see what might be a fit for you.

Image courtesy of Shutterstock

As more and more information floods into the Internet, organizing and making sense of this Big Data becomes more important and more difficult.

New database methods are emerging to help process unstructured data, but IT developers and database deployers also have to figure out how to deal with the world of legacy technology.

For the last 40 years, relational database programs (usually powered by SQL-based management systems) have been the backbone of supplying businesses with organized rows and columns of data. The problem is that these legacy systems may not be able to work together to give businesses the information they need when they need it. Older programs may also have trouble processing data requests over long distances.

A New Way Of Thinking About Databases

A new way of thinking is needed. Over the past decade the push for “not only SQL” or NoSQL database software has provided a pathway for businesses to connect bits and pieces of data from a variety of sources at very rapid speeds across different geographies.

(See also What's Next For Taming Big Data?)

Some businesses are spreading out the workloads using noSQL databases within cloud computing-based networks. Others are approaching the problem still using traditional SQL relational database software - and that’s perfectly OK.

Previous articles in this series (Taming Big Data) discuss the benefits of a noSQL database tool like VMware’s vFabric GemFire. But SQL database software retains a well-established community of tens of thousands of developers and integrators who may be reluctant to move beyond the SQL they know and love. What’s a company to do?

(See also VMware's Cloud-Based GemFire Makes It Easier To Work With Big Data.)

For SQL diehards, VMware’s vFabric SQLFire SQLFire is a distributed SQL database typically used for online transactions. The software is more modern than most traditional relational database management systems.

What Can SQLFire Do For Me?

SQLFire functions and performs much like GemFire under the hood. SQLFire uses GemFire's data grid engine, which lets both programs capture data and then replicate and partition the information "in-memory” on the server. But instead of having to learn GemFire commands and controls, SQLFire has a user interface and programming framework that will be familiar to developers used to programming in a SQL interface and with SQL tools.

Backups are enabled through virtual copies on other connected servers, although data can be stored long-term on disks as needed.

Unlike other embedded databases, SQLFire allows several servers to store replicated and partitioned tables, persist data to disk, communicate directly with other servers and participate in distributed queries.

For traditional IT developers and database deployers, the SQLFire interface makes it easier to write applications and take advantage of GemFire’s underlying noSQL technology. Developers and integrators who know SQL well will have an easy time adapting SQLFire to new projects.

SQLFire is perfect for classic Web transactions, especially where there is a need for fast speeds and a requirement to dig deep into clusters of data.

Business Case For SQLFire

In addition to making SQL developers feel comfortable, SQLFire can work across multiple networks and geographies. This comes in handy when enterprises need information at the moment it becomes available on multiple continents.

For example, a large regional bank in the Northeastern United States collects large amounts of data that helps it maintain its regional and branch offices. The bank also monitors customer transactions at tellers and various ATMs.

Bank management was interested in measuring the different types of transactions being handled at each of type of station, what types of accounts they were accessing and the various times of day the transactions took place.

Historically, the bank could attach an individual database to each branch, but in today's global environment the company decided it needed to measure all of these data points at the same time for each office. The company tested vFabric SQLFire against its own systems and found the existing server took 20 minutes to complete the queries while the SQLFire server completed its task in less than a minute.

Deploying SQLFire In The Enterprise

In the enterprise sQLFire is generally found on inexpensive computer servers in database clusters. A typical use case would find SQLFire helping eliminate potential data bottlenecks in new mobile and Web environments. Another common deployment option for SQLFire is to integrate it with existing traditional databases or analytics programs.

The software can also be interfaced through an API using programming languages such as Java or Spring. SQLFire is also compatible with Java database (JDBC) or ADO.NET.

As companies look for new ways to make data accessible and provide a consistent view of that information, it's important to have tools that suit the needs of all kinds of developers and IT managers.

VMware’s GemFire and SQLFire software are designed to address just those needs - allowing companies to move beyond concerns over speed and scale and tackle Big Data applications head on.

NoSQL databases are well-known for their speed and scalability - useful traits when dealing with the size and complexity of big data and hyper-fast transaction requirements. But one thing they have lacked has been strong data consistency: the ability to ensure that an update to data in one part of the database is immediately propagated to all other parts of the database.

A startup database vendor launched this week is making claims that its database, FoundationDB, finally delivers on the promise of true data consistency for a NoSQL database, without a huge loss of speed or flexibility.

Understanding why this is such a big deal in the Big Data (or any) sector requires a little background on how NoSQL, or non-relational, databases work.

Solving The ACID Test

When talking about relational databases, like PostgreSQL, MariaDB, Oracle and the like, there's one acronym that keeps coming up: ACID. ACID stands for Atomic, Consistent, Isolated and Durable - core aspects that must apply to all data within a relational database. Data is broken down to atomic values (name, address_1, city...) while remaining consistent across the database, isolated from other transactions until the current transaction is finished, and durable in the sense that the data should never be lost.

The infrastructure of a relational database is well-suited to meet the ACID criteria for data: Data is held in tables connected by relational algebra, and transactions are performed in a way that is consistent with ACID principles.

But for non-relational databases, such as Bigtable, MongoDB or Dynamo, ACID has always been sacrificed for other qualities, like speed and scalability.

This tends to freak out some companies, stopping them from moving to NoSQL because they can't give up ACID. Especially the "C," because not having data consistency is a particularly terrifying prospect for companies dealing with financial transactions.

Yet non-relational databases are being used by firms like Amazon and Google every day, with great success. Amazon, in particular, needs to track millions of sales transaction on any given day - how does it get away with inconsistent data?

The short answer is, it has to. The trade-off would be a relational database that could never keep up with the speed and scaling necessary to make a company like Amazon work as it does now. Recall that non-relational databases are structured to sacrifice some aspect of ACID to gain something in return. In the case of Amazon, its non-relational DynamoDB database is willing to apply an "eventually consistent" approach to the data in order to gain speed and uptime for the system when a database server somewhere goes down (though Dynamo can also have strong consistency, an Amazon spokesperson informed us after this story went to press).

Bringing Back Consistency

It's not that having ACID compliance on a NoSQL database is impossible, explained David Rosenthal, one of FoundationDB's co-founders. It's just that most people think that applying ACID to NoSQL systems would come at a huge cost.

That's certainly what Werner Vogels, CTO of Amazon, thought in a 2008 paper that described the company's Dynamo database and it's relationship to consistency. 

Data inconsistency in large-scale reliable distributed systems has to be tolerated for two reasons: improving read and write performance under highly concurrent conditions; and handling partition cases where a majority model would render part of the system unavailable even though the nodes are up and running.

Translation: Requiring ACID on non-relational databases would make that database too slow and inflexible.

For the longest time, everyone using NoSQL systems was resigned to this eventual, or "weak," consistency model. After all, they had money to make and data to analyze. Who cares if consistency was not at the top of the priority list?

It turns out, quite a few people, including the founders of FoundationDB, Rosenthal, Nick Lavezzo and Dave Scherer.

Inside FoundationDB

After a successful start up with Visual Sciences, a technology that's now part of Adobe as the Adobe Insight product, the trio turned to developing another successful project, and hit on the lack of ACID-capable non-relational databases as a goal.

"We weren't satisfied with any of the data guarantees on non-relational systems," Rosenthal explained, even as they understood that the needs of many potential clients would preclude relational systems like MySQL or Oracle because of performance limitations.

Non-relational systems seemed to wear their weak consistency model like a badge of honor, but in the secret origin story of FoundationDB, the team saw weak consistency as a bug, not a feature. "Not having transactional integrity is not a good thing," Rosenthal emphasized.

They're not the only ones. Google's up-and-coming Spanner database, a second-generation distributed database that could ultimately replace the search engine company's Bigtable systems, is being built on the premise that transactional integrity has to be a part of that database, too.

Side Effects Include...

Establishing consistency in transactions within a NoSQL database is worthy news in itself, but the implications extend beyond that core news.

FoundationDB uses a key-value-like storage engine core that's surrounded by layers of whatever data model that's needed, which will in turn enable developers to much more easily code their apps to reach into the FoundationDB. These layers, according to the founders, can't be used on other key-value systems, because without consistent transactions, it would not work.

Also, since data is going to be consistent, applications won't have to be built to "wait" for data to catch up within a given transaction - thus making apps less complex and easier to build.

The best news of all concerns the so-called performance penalty that many in the NoSQL world said will be incurred if ACID was applied to non-relational database systems. According to FoundationDB, performance is hampered by only 10%, which seems a very small price to pay for consistent transactions.

The FoundationDB database, which was launched into public beta on Monday, is available for download now.

Image courtesy of FoundationDB.

While companies of all sizes are struggling with the growth of information overload often referred to as “Big Data,” some IT developers and database deployers are approaching the challenge with a cloud-based service designed to make accessing mass amounts of data faster.

In short, they turn to VMware’s vFabric GemFire.

GemFire is a distributed in-memory data grid database software product that enables data distribution, data replication and partitioning (sharding), cashing data management at the exact moment the information is needed.

(See also What's Next For Taming Big Data.)

While the ability to move data from server to server and replicate it to more than one location has proven invaluable over the last 10 years, today's critical challenge is how can companies manage this data properly.

Over the past decade, GemFire has helped companies:

• Maintain simultaneous data connections over long distances.
• Protect their data from natural and man-made disasters.
• Maintain data reliability and availability, even when server hardware periodically fails.

The software is able to achieve these goals by creating an object-oriented "data fabric" across a server cluster. It accesses copies of data that are stored in various locations as needed. To ensure compatibility with the latest cloud configurations, the management platform can spread the data across many virtual machines and GemFire servers to manage application objects.

But what does that mean in the real world? To find out, it helps to look at how vFabric GemFire is already working in key industrial applications, how it can be developed for new projects and how it can be deployed in a business network.

Passing Military Grade

Keeping connected across town or around the globe is never more important than when national security is on the line. So when the U.S. Defense Information Systems Agency (DISA) needed to deal with up-to-the-minute information and awareness of military actions wherever they occur, the agency chose vFabric GemFire.

GemFire provided speed and the ability to easily increase and decrease the size of projects, but also a management tool that orchestrates data delivery from the back-end data stores to the consuming applications.

Since 2007, DISA has used GemFire for managing massive amounts of data for the various government agencies it supports, including U.S. military commands, joint task forces and the Pentagon. And because GemFire allows for a consistent view of data across all geographies and in different clusters, the military has reliable event notification, continuous querying, parallel execution, high throughput, low latency, high scalability, continuous availability and WAN distribution

South America Calling

GemFire’s expertise at the middle data tier delivers reliability and critical data redundancy that keeps the information up to date even if one part of the network goes offline.

Take the case of a large telecommunications company in South America that sells prepaid phone cards via kiosks. The telecom uses GemFire to enable the sale and provisioning of pre-paid cards even when disconnected from the network. Because the country’s infrastructure is not 100% reliable, sometimes network data is not updated for several hours at a time and customers might not be able to use their cards. To overcome this obstacle, the telecom uses GemFire's distributed databases to maintain up-to-the-minute information.

vFabric GemFire was the optimal choice for managing a distributed database in this environment because it automatically recognizes systems and moves data around so that it remains accessible even on unreliable networks.

As VMware product line marketing manager Blake Connell put it, “vFabric GemFire automatically spreads the data over a wide network and accommodates network disruptions.

Capturing GemFire In The Enterprise

vFabric GemFire is best suited for new Big Data projects that require NoSQL - or distributed unstructured data - models.

GemFire is well-designed for latency-sensitive applications such as virtualized environments that may require interrupt-moderation or interrupt-throttling - industry terms that IT developers and database deployers use when building a system that potentially doesn't take well to lags in data flow or processing.

Because GemFire is designed for data distribution, data replication, caching and data management, it has special requirements. For example, GemFire suggests enabling hyperthreading and keeping at least 50% of the server’s memory space available.

Configuring GemFire servers and regions is optimally done with the Spring object-oriented programming framework. This allows developers to centralize application service configuration instead of having to deal with Spring context configuration plus a separate cache.xml file.

For those working with structured data and who are knowledgable in SQL, VMware offers a related product called SQLFire. SQLFire is a distributed SQL data-management platform. SQLFire will look familiar to SQL developers thanks to a similar interface and programming framework, and it allows the management of "not only SQL" databases much the way GemFire does.

Look for more information on the benefits of SQLFire in an upcoming ReadWrite post.

Imagine you are walking in London and want to book a trip to New York at the last minute: You hop online to search for a hotel. The smartphone you are holding helps you find an establishment that looks appealing. You tap a few buttons to send your credit card information.

Unbeknownst to you, however, there is only one available room left at this hotel for the night you need. You and and another traveler, from Brazil perhaps, are vying for the same space. That person is trying to make the reservation from a laptop at his office.

• Could these two transactions be handled at the exact same time?
• Can the hotel see a consistent view of this globally distributed data?
• Could the transactions be handled in a way that the hotel owner can identify where each request originated from?
• Could the hotel distinguish on which device the purchase was made and any other personal information required?

If the two transactions were being handled by a traditional SQL database with a siloed network configuration and little ability to scale on demand or access data in real time, the short answer would be, “No.”

As Jerry Chen, VP of VMware’s Cloud and Application Services noted recently: “The monolithic database of the past cannot meet the needs of modern applications.” What is needed, he added, are designs that support high-availability low-latency applications.

The Influence Of Big Data

Your experience with booking a room is a perfect example of how large amounts of important, fast-moving information - commonly known as Big Data - is creating problems and opportunities for customers and the companies that serve them.

While modern technologies provide faster access to higher volumes of both structured and unstructured information at any time of the day, making sense of the bits and pieces of unrelated data has required a radical shift in the way we organize networks and the Internet. It’s a challenge for even seasoned IT developers and database deployers.

In addition, real-world events such as the Arab Spring and the earthquake and tsunami in Japan have challenged engineers and developers to break through existing constraints and work wonders in hyperconnectivity using non-traditional methods.

Historically, companies sorted customer and transactional data in neat rows and columns of databases. These databases were built and managed by Structured Query Language (SQL) and stored in isolated, or siloed, networks.

But traditional, hard-disk-drive-based database products forged on SQL can no longer handle the deluge of Big Data in a timely fashion. The era of Big Data calls for new strategies and new approaches to queries.

The limitations of traditional SQL-based databases have prompted a slew of innovative developments in this direction starting with NoSQL distributed database models. These types of software platforms have caught the eye of decision-makers in the financial services, e-commerce, government and R&D communities.

NoSQL has the capacity to answer to their need to process Big Data sets as they happen. However, many NoSQL deployments still fail to cope with scenarios like the one detailed above because they cannot provide consistent speed or scale. Additionally, many NoSQL offerings lack maturity when dealing with data that needs to span multiple datacenters across vast distances.

Other Innovations Add Support

To keep up with the true volume and velocity of information from multiple sources, innovations in cloud computing are being paired with high-performance “in-memory” storage on database appliances created to address the obstacles of managing Big Data.

While cloud-based systems have helped develop online computing and software service delivery systems, IT developers and database deployers have adopted the cloud to take advantage of the ability to scale to globally distributed marketplace. Meanwhile, the addition of solid-state memory drives to storage devices has given companies an extra layer of quickly accessible data. These configurations allow for more rapid database queries and transactions - and thus the ability to deal with much larger data sets in real time.

But the raw technology is only part of the equation. No matter how powerful the exterior influences, IT developers and database deployers will always need tool sets that help them to design for their current Big Data database needs (such as our hotel booking example) as well as prepare for a future where Big Data needs to be addressed at multiple levels.

The tools must be customizable and easy to use - no matter the developer’s existing training. These offerings must also be easy to manage and easy to connect with new technology trends.

VMware’s GemFire and SQLFire software are designed to address just those needs of developers and deployers.

GemFire is a database software product that allows for data distribution, data replication, caching and data management at the exact moment the information is needed.

SQLFire is a distributed data management platform which functions and performs much like GemFire, but has a user interface and programming framework familiar to developers who are more comfortable programming in SQL interface and tools.

Both GemFire and SQLFire help companies move beyond concerns with speed and scale and tackle Big Data applications head on.

Image courtesy of Shutterstock.

IBM spends over $1.5B every quarter in research and development (R&D) expenses. SAP? Closer to $700M. Oracle? Assuming we don't count the tens of billions it spends buying other companies, its actual quarterly R&D budget comes in just over $1B - around $900M. Microsoft, which has printed billions of dollars in profit each quarter for eons, spends more than them all, topping $10B each year.

And yet not one of these companies is responsible for the biggest advances in enterprise technology in the past decade.  Cloud computing, Big Data, mobile... they're all being invented elsewhere, not by the enterprise behemoths.

Maybe they're doing it wrong?

Lots Of R, Little D

Take the cloud, for example. Microsoft claims to invest 90% of its R&D budget on cloud computing, but it is Amazon, Microsoft's penny-pinching, book-retailing neighbor, that sets the terms for innovation in cloud computing. Amazon launched EC2 back in 2006, when it was spending a measly $132M or so each quarter on R&D. 

Even if we dismiss Amazon, where else are we seeing other cool advances in cloud computing? Netflix, for one, which just released Janitor Monkey to help Amazon Web Services (AWS) users dispose of their unused AWS resources, and the video company previously released Chaos Monkey, which helps enterprises plan for and architect around cloud failure.

Notice the word I used? "Released." It means these tools were open sourced, not put out for sale. That's how innovation seems to happen in the 21st Century.

In large part innovation comes with an open-source license because it's a by-product of businesses that heavily rely on technology, but don't actually sell technology.  It's 'trickle-down innovation' from the web business community.

VMware's Brad Hedlund spotlighted this trend back in 2011, when the enterprise awoke to discover it had problems that the web giants had already solved:

As properties such as Yahoo!, Google, Facebook, Amazon became great successes, their architects and software engineers realized that they had moved mountains...The tremendous problems of efficiently running large scale applications on low cost infrastructure had been solved... At the very same time, enterprise IT begins to encounter some of the very same problems solved by the large web provider, such as scalable data warehousing and analytics (so called “Big Data”). Additionally, the software driven distributed systems that solve problems of infrastructure efficiency and management at very large scale could also be applied to infrastructure at a smaller enterprise IT scale (why not?). And finally, the cost savings of an application infrastructure designed to operate on low cost commodity hardware can be realized at any scale, large web or enterprise IT.

Filling The R&D Gap

Companies like Cloudera, DataStax and others stepped into this gap, taking the open source (or, in the case of some of Google's research, open knowledge) projects from the web and applying them to the enterprise in the form of Hadoop, Storm, NoSQL databases, etc.  All of it developed at a comparative pittance to enterprise incumbents' R&D budgets.  All of it available free of charge on commodity hardware. 

As an industry, we're richer for such open source innovation.  Ironically, so are the enterprise IT vendors, who are investing tens to hundreds of millions of dollars in Hadoop and other open source data and cloud innovations, even as they continue to sink tens of billions of dollars into their homegrown R&D. Maybe it's time for them to reevaluate how they do R&D.  Maybe, like Facebook or Twitter, they should release their R&D on GitHub as open-source code.  At the least they could, like Google, centralize their research on the web, making it easily available to all.

Maybe, just maybe, they'd realize that it doesn't actually matter how much money a company spends on R&D.  What matters is whether it can execute and turn ideas into winning products, as Brad Reed argues, and whether it can help foster community around promising open source efforts.  

This is what new enterprise IT - Facebook, Twitter, Google and Yahoo - demonstrates.  It remains to be seen if IBM, Microsoft and other traditional IT vendors are paying attention.  Unless they do, they'll lose relevance as a new breed of innovative startups emerge to claim the strategic largesse of CIOs' budgets.