As more and more information floods into the Internet, organizing and making sense of this Big Data becomes more important and more difficult.

New database methods are emerging to help process unstructured data, but IT developers and database deployers also have to figure out how to deal with the world of legacy technology.

For the last 40 years, relational database programs (usually powered by SQL-based management systems) have been the backbone of supplying businesses with organized rows and columns of data. The problem is that these legacy systems may not be able to work together to give businesses the information they need when they need it. Older programs may also have trouble processing data requests over long distances.

A New Way Of Thinking About Databases

A new way of thinking is needed. Over the past decade the push for “not only SQL” or NoSQL database software has provided a pathway for businesses to connect bits and pieces of data from a variety of sources at very rapid speeds across different geographies.

(See also What's Next For Taming Big Data?)

Some businesses are spreading out the workloads using noSQL databases within cloud computing-based networks. Others are approaching the problem still using traditional SQL relational database software - and that’s perfectly OK.

Previous articles in this series (Taming Big Data) discuss the benefits of a noSQL database tool like VMware’s vFabric GemFire. But SQL database software retains a well-established community of tens of thousands of developers and integrators who may be reluctant to move beyond the SQL they know and love. What’s a company to do?

(See also VMware's Cloud-Based GemFire Makes It Easier To Work With Big Data.)

For SQL diehards, VMware’s vFabric SQLFire SQLFire is a distributed SQL database typically used for online transactions. The software is more modern than most traditional relational database management systems.

What Can SQLFire Do For Me?

SQLFire functions and performs much like GemFire under the hood. SQLFire uses GemFire's data grid engine, which lets both programs capture data and then replicate and partition the information "in-memory” on the server. But instead of having to learn GemFire commands and controls, SQLFire has a user interface and programming framework that will be familiar to developers used to programming in a SQL interface and with SQL tools.

Backups are enabled through virtual copies on other connected servers, although data can be stored long-term on disks as needed.

Unlike other embedded databases, SQLFire allows several servers to store replicated and partitioned tables, persist data to disk, communicate directly with other servers and participate in distributed queries.

For traditional IT developers and database deployers, the SQLFire interface makes it easier to write applications and take advantage of GemFire’s underlying noSQL technology. Developers and integrators who know SQL well will have an easy time adapting SQLFire to new projects.

SQLFire is perfect for classic Web transactions, especially where there is a need for fast speeds and a requirement to dig deep into clusters of data.

Business Case For SQLFire

In addition to making SQL developers feel comfortable, SQLFire can work across multiple networks and geographies. This comes in handy when enterprises need information at the moment it becomes available on multiple continents.

For example, a large regional bank in the Northeastern United States collects large amounts of data that helps it maintain its regional and branch offices. The bank also monitors customer transactions at tellers and various ATMs.

Bank management was interested in measuring the different types of transactions being handled at each of type of station, what types of accounts they were accessing and the various times of day the transactions took place.

Historically, the bank could attach an individual database to each branch, but in today's global environment the company decided it needed to measure all of these data points at the same time for each office. The company tested vFabric SQLFire against its own systems and found the existing server took 20 minutes to complete the queries while the SQLFire server completed its task in less than a minute.

Deploying SQLFire In The Enterprise

In the enterprise sQLFire is generally found on inexpensive computer servers in database clusters. A typical use case would find SQLFire helping eliminate potential data bottlenecks in new mobile and Web environments. Another common deployment option for SQLFire is to integrate it with existing traditional databases or analytics programs.

The software can also be interfaced through an API using programming languages such as Java or Spring. SQLFire is also compatible with Java database (JDBC) or ADO.NET.

As companies look for new ways to make data accessible and provide a consistent view of that information, it's important to have tools that suit the needs of all kinds of developers and IT managers.

VMware’s GemFire and SQLFire software are designed to address just those needs - allowing companies to move beyond concerns over speed and scale and tackle Big Data applications head on.

In 2013, traditional data centers will begin to lose their dominant status within the data-management food chain. They will increasingly be replaced by big-data software and lower-cost, ARM-based systems-on-chips. 

When thinking about the future of data centers, the problem is one of scale. For the past few decades, relational databases and the attendant hardware that runs them have been able to manage pretty much anything a company could throw at them, but those days are coming to an end.

When Relational Ruled The Land

In the beginning, and for the first 20 years or so, data was heavily transactional, and was managed in discrete and very secure ways. Speed was less important than making sure the data was safe as houses.

In the 1990s, data began to be used in a slightly different way, as comapnies placed analytical demands on the data being gathered. Instead of being retreived in discrete packages, data became as a strategic asset to be analyzed, leading to the disciplines of business intelligence. Databases grew into massive data warehouses, and parallel querying arose as the only way to effectively manage the staggering workloads placed on information technology.

Through the early years of electronic data, growth in the volume of data may have been rapid, but data tools and infrastructure were pretty much able to keep pace.

That's not so true anymore. Software soon will not be able to cope with the overwhelming volume of data being generated, says Mike Hoskins, chief technology officer of Pervasive Software. What's coming is a real break in how data is managed.

Breaking The Old Model

To give an idea of what kind of scale we're talking about, Hoskins points to U.S. retailer Wal-Mart's estimated 1-petabyte data store.

"That's the accumulation of 40 years of Wal-Mart sized business," he said. "Facebook? Facebook generates that much data in a week."

There's always a collection of data behind each transaction. But in e-commerce today, a customer can be clicking around quite a bit before buying, which leads to useful data sets tens, hundreds or thousands of times larger than "so-and-so bought widget X with credit card Y." Add the fact that the machines handling these activities are also recording machine-to-machine transactions, and the data workload explodes beyond the capacity of any traditional data center. 

"We are reaching the end of the useful life" of our data centers, Hoskins said. "The bottom line is, it's a death march."

Even if conventional software could manage this explosion, no company could afford it. Not to mention the energy costs invovled in buying, running and cooling the hardware.

Indeed, it is innovation in hardware that's going to provide the evolutionary break that Big Data requires. Servers with ARM-based processors, which absorb something like 20 times less power than Intel-based processors, are the next wave in data center infrastructure. Lower power requirements, after all, mean less resistance and less heat. Less heat means less money wasted on cooling and the ability to compress ARM-based systems closer together.

As energy and general hardware costs coem down, hardware is lined up to take care of the new data workloads of this new massive scale of data.

First Hardware - Then Software

On the software side, Big Data will increasingly be handled by Hadoop systems that can store data and manage and analyze Facebook-scale loads.

If you're wondering why this is supposed to be big news, think about it this way: Relational databases have been handling data of all shapes and sizes for decades, and now there will be a certain level of data that the traditional data center architecture will simply be unable to handle. It's the first stratification of data management. 

On one level of data management, relational databases will still be around, supporting smaller, less complex and more tactical workloads. But on this new level, whole new architectures will be created to deal with this scale.

What's The Data Center Endgame?

This half-way approach is not the end game, though.

Eventually, Hoskins believes, tools will be built into the Hadoop framework that will enable data managers to run applications and analysis right where the data lives, inside the Hadoop clusters.

It's no accident then that the latest iteration of one of Hadoop's core components - MapReduce 2.0, code-named YARN - includes the beginnings of a framework that will let developers build exactly those kinds of tools inside Hadoop. This is something that the VP of Apache Hadoop Arun Murthy confirmed to me early this year at the Strata Conference in Santa Clara, California. When the YARN application framework is robust enough, Hadoop will be able to let developers code those applications.

This will be the new way of working with data as it gets too big for relational databases to handle: a new architecture of low-cost, low-power servers that will keep applications and data as close to each other as possible, in order to maximize efficiency and speed.

"Relational database technology has had a good run," Hoskins said. But the days of the relational database being a part of every data solution are fading fast, as a new kind of data center becomes the new sheriff in town.

Image courtesy of Shutterstock.

It's been just under seven months since Red Hat became the world's first $1 billion open-source company. Now the question is who will follow suit and become the next open source company to hit this milestone?

Before sorting through the list of likely candidates, it's important to define what we're talking about when we say "$1  billion company." Primarily, we're talking actual annual revenue, not market valuation. That's a key difference, since under the valuation definition, Red Hat would not have been the first billion-dollar open source company; MySQL AB would have had that distinction, when Sun Microsystems bought it for $1 billion in 2008.

And, when we're talking open source, the idea is a company that bases its business on the sale and distribution of open-source software. Things get a bit tricky here, since many companies could be assigned the "open source" moniker - and some might argue against it. This is particularly true of companies that follow the so-called "open core model," which typically release a free version of the software as a "community" edition as well as a more feature-rich but closed edition that's available for a licensing fee or support subscription.

Red Hat is not open core, because even though it sells a "closed" enterprise version, Red Hat Enterprise Linux, and helps distribute a free version of the product, Fedora, the truth is that the source code for Red Hat Enterprise Linux is still freely available, so one can, if desired, use RHEL without support.

While it would be possible to reject open-core model companies, the fact is that this would sharply limit the field of potential candidates for the next $1 billion open source company, so I'm going to keep them in mind and note them as such.

Our three top candidates - all private companies - definitely vary in the kind of software they offer, but each one has an opportunity to be the next open-source company that crosses the billion-dollar line.

The Database Candidate

With all the hullaballoo about Big Data (see next candidate), it's important not to forget there are still a lot of fast and flexible offerings in database-land that still follow the old rules of managing data. Big data doesn't mean that relational databases are dead; in fact, such databases are more important then ever.

EnterpriseDB is in the business of distributing commercial products built on one such open source database, PostgreSQL, a direct competitor to the MySQL mentioned earlier that was snatched up by Sun and subsequently acquired by Oracle.

On its own, PostgreSQL (or Postgres as geeks call it), is hot right now. The open source Postgres is getting a lot of new features and the pace of development is perceived as faster than its MySQL counterpart. Plus, there is a definite interest in Postgres driven by a movement away from MySQL. While MySQL is still very powerful software, there is a lot of tension in the marketplace about Oracle's stewardship of that project, since Oracle dropped the ball on several of the open source projects it acquired from Sun and many feel that eventually MySQL will suffer from the same problem, with Oracle either not pursuing development as much as it could or (as some fear) closing MySQL altogether.

While MySQL's loss is PostgreSQL's gain, Postgres is getting attention on its own merits. EnterpriseDB, which offers an enterprise version known as Postgres Plus, is making big inroads with customers like Sony Entertainment.

Data on a smaller scale will always be more prevalent than Big Data, which means Postgres and the vendors that package it have a nice, wide market into which to play. That gives EnterpriseDB a clear path to becoming the first $1 billion open source database company.

The Big Data Candidate

Nevertheless, you can't ignore the hype... Big data is a big driver for commerce right now, and any big data vendor has to be considered for billion-dollar revenue status.

The technology that's leading the big data charge is the open source data framework known as Apache Hadoop. Hadoop is not a database, but rather a distributed storage tool that enables programmers devise jobs in Java code to search for desired information - without formal database structure.

Hadoop is not the only big data tech, but right now it's the top technology in the sector. And right now the biggest Hadoop company is Cloudera.

Though MapR and Hortonworks are strong Hadoop contenders, Cloudera has managed to establish an early lead in this sector. Cloudera has developed its own tools to make it easier to seek information from Hadoop stores and is building a rich partner ecosystem to enable different applications of Hadoop-stored data.

If Hadoop continues its rocket-ship rise, it's not hard to envision Cloudera becoming the first $1 billion open source Big Data company.

The Cloud Computing Candidate

In the world of open source cloud computing, OpenStack is the name that gets the most attention, thanks to the combined marketing prowess of Rackspace, HP, Dell and Red Hat, just to name a few.

But OpenStack is not a company or even a product - it's a project. Eventually, one of the OpenStack vendors may come up with a commercial product and run with it, but if its someone like Red Hat or HP, they've already made their mark. Piston Cloud or SUSE may also succeed as billion-dollar candidates, but right now the OpenStack system is a free-for-all and no one seems to have a commanding lead.

So I'm betting on an open source cloud player that's not even in the OpenStack ecosystem: Eucalyptus.

Eucalyptus gets the nod because of OpenStack's lack of product and the fact the other open product in this space, CloudStack, has Citrix as its flagship commercial vendor and it's not really an open source company. Eucalyptus also gets marks for being very well connected to the Amazon Web Services ecosystem. While you may argue that sticking to AWS frameworks is really not all that open (in fact, I have made that very argument), there's no denying that AWS is killing in this space and a lot of customers interested in private or hybrid cloud computing are very content to keep swimming in Amazon's pond.

But there is more to Eucalyptus than being in the right place at the right time. It has got a whip-smart team, led by Marten Mickos, who - in one of those "small world" things - was the CEO of MySQL AB, the same company Oracle now has in its portfolio and was valued at $1 billion upon acquisition.

With all of these stars aligned, Eucalyptus could soon be the first billion-dollar open source cloud computing company.

Sure, other candidates may be in the running for the next billion-dollar open source company. And there is always a chance that someone new, like Big Switch Networks and its virtual networking systems, could come out of nowhere and snatch the title.

But these three candidates, based on what I see right now in the open source world, are presently the most likely contenders for becoming the next $1 billion open source company.

Lead image courtesy of Shutterstock.

(Click to enlarge the screen. There is a "before" picture posted on TechReview's site too.)

Of course, by October things were a bit different in the Twitterverse, as on the streets of New York and elsewhere. The above screen shot shows centers of influence from Huffington Post and Keith Olbermann, with smaller nodes from the New York Post, CNN and the Associated Press. Where is the coverage by the Grey Lady? Too small to see with my eyes. That in itself shows you how the message was spread and who was relevant and who wasn't. Fascinating.

The Massachusetts Institute of Technology may be the birthplace of the American geek. Within MIT, its Media Lab drills down to the heart of the next wave of technology from creating buildings with 3D printing to prosthetic limbs to gesture-based user interfaces. For instance, the MIT Media Lab was where the idea for the technology seen in the movie Minority Report originated.

The unofficial motto of the MIT Media Lab is "demo or die." It is akin to the classic academic model of "publish or perish," except that students and faculty at the Media Lab are encouraged to actually create the products they are thinking up, as opposed to pontificating upon them in research papers. See below to check out some of the amazing waves of technology that will be bursting out of the Media Lab in the future.

Interactive Robots & Holographic Imaging

There is a little orange robot called Combusto. It can show emotion and has a wide array of movement. Think Wall-E with fur. Combusto is powered with an Android smartphone that has a unique chipboard in it that helps determines Combusto's functions.

Right next to the robot lab, there is a team working on holographic 3D imaging. Another line from Star Wars comes to mind: "Help me Obi-Wan Kenobi, you are my only hope." One of the goals for the holographic imaging team would be to make it feasible to make those Star Wars-style holographs a reality.

Throughout the Media Lab, students, researchers and faculty are using the Microsoft Kinect for gesture-based moving and tracking. The Kinect API is one of the most innovative consumer technologies to hit the market in a long time and its affordability makes for perfect use in research settings.

Tangible Media

If you put the ideas of holographic images together with the notion of tangible media, you get the computer that Tom Cruise uses in Minority Report. The company that is trying to make that vision a reality is called Oblong, and it originated at the Media Lab.

Tangible media is the idea of being able to manipulate a user interface either through gesture or interacting with a physical object connected through the Internet. This can also be done through Kinect, although the department also has a series of infrared cameras that can track movement and intention with more precision. The cameras are used to localize an item (say, a person) in a 3D space.

Tangible media is described as where the input is also the output. Say I have a gesture-based camera watching my movements. If I am using building blocks in physical space, those same building blocks should move in the digital space as well.

Capacitive touch, which is perhaps the biggest breakthrough in the smartphone revolution, also has origins with projects in the media labs. Indirectly, the iPhone and touch-based devices have roots at MIT.

Robotic Opera, 3D Printing & More

It is hard to keep track of everything that is happening at the MIT Media Lab without being an insider. There are teams working on prosthetic limbs that reduce the impact of missing legs and let a person move around normally. There is a storytelling center looking at new ways to present digital media. There is a camera center that is working on how best to utilize the powerful cameras now attached to every person's hip via cellphone. There are teams working on creating different densities of material coming through a 3D printer that can make the building blocks of structures with concrete. The innovators behind Rock Band came from the MIT Media Lab. These days, the music group is working on robotic operas.

The MIT Media Lab is majority funded by sponsors or "members" that are companies working on specific functions. For instance, DirecTV helps fund some of the interactive television projects that are being worked on. The building is designed to be an open center of collaboration where everybody can see what the others are working on. The idea is to get designers and scientists together to create technology that will have an impact on the world.

Map source: Ben Legler, University of Washington and Derick Poindexter, Appalachian State University

The two are completely different efforts, but the idea is to extend botanical knowledge through various Web-based materials and allow scientists and students to collaborate online. A herbarium is typically a huge collection of dried plants maintained by large universities and botanical gardens - two of the largest are found at the Bronx, NY and St. Louis, Missouri. To give you an idea of the size of these things, the St. Louis collection contains more than 5 million plants and is housed in several buildings. There are several hundred others located all over the country as you can see by the map above.

Led by Mary Barkworth, a professor at the Utah State University, the project aims to coordinate all of these heretofore independent herbaria efforts around the country. The virtual edition has just gotten started, and the researchers are figuring out common data markups and trying to catalog the individual plant species that each institution has on file, which Barkworth estimates is in the several millions. Part of the problem is that the plant collections go offline, as universities change their funding and science emphasis, and are either tossed aside or dispersed to interested collectors. So, tracking these herbaria down isn't a simple task. Another problem is that these collections aren't static: researchers are continually adding to their collections.

Eventually, the entire catalog will be imaged and placed on the public Internet so that researchers from anywhere can find a particular leaf, fern or seed. Right now there is the Index Herbarium maintained by the Bronx NY Botanical Garden here, but it is more a collection of pointers to institutions, rather than a database of the actual plant materials itself.

The Open Science network is an entirely different effort, but also presented its progress at the biology conference this week. Here more than 40 academic institutions are trying to help science educators improve their curricula with all sorts of virtual tours: you can take a recorded video walk in the woods with a noted bio prof and hear him provide commentary and descriptions of plant life, for example. It aims to make science more hands-on and interactive and engaging. There are links to lesson plans, lectures, and summer field courses that students can take.

The Web world is nothing if not Darwinian: it's survival of the fittest and products need to evolve with the times. Some Web products fly and some don't. Those that don't fly either die out, or evolve into something new. The latter is what happened to Google Base, which in 2011 is a shadow of its former self - and is even about to lose its API. It did however spin off a more successful offering, in the form of the Google Merchant Center for retailers. In this post, we look back on the initial vision for Google Base and then analyze what it actually evolved into.

How Google Base Started Out in 2005

For those of you whose memories don't go back as far as Web 2.0, Google Base is an online database that launched in November 2005. At that time it enabled you to upload any type of information, along with "attributes" that described it further. Early examples of Google Base content included reviews, events, products and jobs. Some - but far from all - of the data was then surfaced in Google properties like search, Froogle and Local.

A snapshot of Google Base homepage, circa January 2006 (source: Wayback Machine)

Despite a clunky user interface and initial mixed reviews, Google Base was a promising product in late 2005 and early 2006 because of its potential to better structure Web content. At the time Google appeared to be aiming to use the product to improve its main search engine, but also to create other Google properties in verticals like e-commerce and classified ads. So there was a lot of talk at the time about it challenging eBay and Craigslist.

This is how I summarized Google Base back in November 2005:

Google Base may not be pretty to look at and it may be a centralized database, but the potential is there to turn it into a hugely valuable directory of structured content. Plus if they add APIs and start aggregating outside RSS feeds, then they could easily extend Google Base and remove the issues around it being a 'walled garden'.

In about April 2006, Google began to expand the number of vertical niches it covered in Google Base. The product was expanding, but how good was the data it was receiving and was there any sign of it being productively used?

Google Base in 2011 & its More Successful Spin-off

Fast forward five years to April 2011 and Google Base still exists (and incidentally still labeled a "beta" product!), but it's been scaled back in two big ways.

Firstly, commercial product data has been separated out into a new Google property called Google Merchant Center, which launched in September 2009. Google Base is now restricted to "non-product data." Data in the Merchant Center is being used in Google Commerce Search, the third generation of which was announced last week. As the name suggests, Google Commerce Search is a search engine for retail - a market which Google obviously sees huge potential in.

The other big difference between the 2011 Google Base and its 2005/2006 version: there's no Web frontend for people to search the data. Whether Google ever even wanted to make Google Base into a long-term frontend product is debatable, although what's not debatable is that it was a user interface mess from the start.

What Happened to the Structured Content Dream?

Looking at how Google Base evolved, one has to ask: was retail data the only valuable thing that came out of Google Base? Did the grand dream of all types of structured content never pan out?

The main problem with structured content has always been that you can't rely on people to upload reliable data for a variety of reasons ranging from motivation to accuracy. As I wisecracked back in 2005, "Google could in fact be building the world's largest database of structured shite."

That's not to say that structured content isn't alive and well. But nowadays the main initiative around this is the W3C's Linked Data, which is getting good uptake from governments and commercial companies alike.

Google is obviously still innovating with new types of data, structured and otherwise, but it no longer views Google Base as a good source of that data. Or at least, not good enough to actively foster - because it's about to deprecate the Google Base API.

Google Base to Lose its API

In December last year, Google announced that it is deprecating the Google Base Data API and replacing it with two new APIs: the Content API for Shopping and the Search API for Shopping. The Google Base Data API will be "fully retired" on June 1, 2011.

What this means for Google Base is unclear, but it's obvious that Google is far more interested in developing the shopping spin-off, Merchant Center. It appears to be giving up on actively developing its database for other types of structured content. In an FAQ about the API deprecation, Google stated:

"The new Shopping APIs won't support use cases that require uploading & searching data for non-shopping applications like Real Estate, Events, Jobs, Activities and other custom structured content applications."

While Google Base lives on, it's a much diluted product. Retail data was clearly the only valuable user-generated content that Google got out of it. Let us know in the comments whether you've used Google Base, in the past or now. What are your thoughts on its evolution and future?

Photo credit: Wikipedia

This year we've chosen 5 products by semantically charged startups and 5 implementations by large organizations. The startups represent the cutting edge of Semantic Web. Each has made an impact on the Internet this year, with user growth and innovation. The organizations we've selected - which include Facebook, Google and the BBC - offered the best examples of large scale deployment of semantic technology.

A note on terminology:we are using 'Semantic Web' and 'Semantic technology' somewhat interchangeably, although many people believe that the term Semantic Web (upper case) should only be applied to W3C-approved technologies such as RDF and SPARQL. The fact is that a good portion of our top 10 use technologies that are either not approved by the W3C (the Web's governing body, led by Sir Tim Berners-Lee), or they've been tweaked in some way - for example, Facebook's use of RDFa. So we've chosen to use the term 'Semantic Web' in its broader, more inclusive, sense. In a nutshell, these are products that add meaning and context to data.

Here then is our list of the top 10 Semantic Web products or implementations of 2010 (in no particular order).

Freebase

In July Google acquired one of the leading Semantic Web companies, Metaweb. Metaweb runs Freebase, an open, semantically marked up database of information. It looks similar to Wikipedia, but Freebase is all about structured data and what you can do with it.

Google already had a relationship with Freebase, pulling in its information to provide intelligent search results within Google News. With the acquisition of Metaweb, Google can now leverage the company's tools and data even further, especially within basic Web search results.

Freebase was one of our top 10 Semantic Web products last year and being acquired by Google validates its potential.

GetGlue

This year was a turning point for GetGlue, the service where users "check in" to watching TV shows, reading books, listening to music and more. Last November, GetGlue changed its branding and launched a new website. It changed almost overnight from a geeky browser add-on called Blue Organizer to a destination website called GetGlue. Mobile applications followed soon after, enabling its users to interact with GetGlue while watching TV or at an entertainment venue.

The changes have been good for GetGlue. It's experienced strong growth this year, reaching over 600,000 users by the end of September.

Disclosure: GetGlue's founder and CEO, Alex Iskold, used to be a regular contributer to RWW.

Flipboard

The launch of the iPad in 2010 triggered a new round of innovation in the startup community. Few startups utilized the touchscreen UI to create a unique user experience more than Flipboard, a magazine reading application built specifically for the iPad.

It turns out that Flipboard isn't just a pretty face, it's also using Semantic technologies.

In July, Flipboard acquired semantic technology startup Ellerdale, whose intelligent data-parsing algorithms had previously been used to create a real-time search engine and trends tracker. Ellerdale's technology was used by Flipboard to design a more personalized real-time experience - determining what social updates are important to you and presenting them in its now familiar magazine-like format.

Hunch

Hunch started out as a Q&A service, but in August it re-positioned as a personalization service. It's a recommendation engine that shows you movies you want to see, books you want to read, vacation destinations you want to go to, and much more. The company is on a mission to "map every person on the Internet to every object on the Internet, be that a product, a service, or a person."

Co-founder Caterina Fake told us in October that Hunch uses a decision tree model, as an alternative to search, to provide more personalized information to users.

Apture

Apture is a semantic contextual search service which continues to iterate strongly (it made our top 10 list last year, too). In August, Apture launched Apture Highlights, a plug-in that allows you to dive deep into any topic you discover on almost any page around the web.

When we first noticed Apture several years ago, it was a service that required publishers to load up linked pop-up widgets with multimedia of their own choosing. The company removed that barrier to entry with its August release. Everything is now automated and it's available almost everywhere. Indeed we liked it so much, we started using Apture on ReadWriteWeb (there is no commercial relationship, we just think the product adds to our site's user experience).

Facebook

Arguably the biggest Semantic Web news of the year came in April, when Facebook announced a large-scale new platform called the Open Graph. The stated goal of the Open Graph protocol was to enable publishers to "integrate [their] Web pages into the social graph." Essentially, each web page can now become an 'object' in Facebook's social graph (which is Facebook's term for how people connect to each other in its network). This means that pages can be referenced and connected across social network user profiles, blog posts, search results, Facebook's News Feed, and more.

The Open Graph is a wide-ranging platform which includes features such as 'Like' buttons and publisher plug-ins. It also includes a simple, RDF-based markup. This requires publishers to include at least 4 metadata properties in each object: title, type, image, URL. There are a few additional properties which may be optionally added, such as site_name and description.

See also: Facebook Open Graph: The Definitive Guide For Publishers, Users and Competitors

Google Squared

The holy grail in web search technology is to be able to ask a simple question, in natural language, and get a simple answer. In May, Google announced that Google Squared was coming to its search results. Google Squared, which launched in 2009, adds additional information to search results.

The functionality was added to Google's traditional search results in two ways. Firstly, simple queries such as Catherine Zeta-Jones' date of birth elicited useful data within the search results:

By clicking "show sources" on the Squared-provided result, a list of sources appears showing you how Google arrived at this answer.

Secondly, Google Squared is being used to provide a new feature in Google's sidebar (another innovation by the search giant in 2010): "Something different". This feature provides a list of related searches that may be of interest, determined by looking at your current search term.

This year Google also reported strong growth in its Rich Snippets feature, which adds extra information to Google search results too - in this case, data like review ratings.

Best Buy

One of the themes of 2010 was the increasing usage of Semantic Web technologies by large commercial companies like Facebook and Google. Leading U.S. retailer, Best Buy, was another large company to impress in 2010 with its adoption of semantic technologies. Specifically, Best Buy used a Semantic Web markup language called RDFa to add semantics to its webpages.

Jay Myers, Lead Web Development Engineer at BestBuy.com, told ReadWriteWeb in an interview earlier this year that the primary goal of using semantic technologies was to increase the visibility of Best Buy's products and services. With data such as store name, address, store hours and GEO data being marked up using RDFa, search engines are now able to identify each of those data components more easily and put them into context. The use of semantic technology, Myers told us, led to increased traffic and better service to its customers.

Data.gov.uk

In January, Data.gov.uk launched to make non-personal data held by the U.K. government available for software developers. It arrived six months after the U.S. government launched its Data.gov site, but from the start the U.K. site had more than three times as much data. At launch, Data.gov.uk had nearly 3,000 data sets available for developers to build mashups with. By the end of the year, that had increased to over 4,600.

Data.gov.uk was one of the highlights of the year in Linked Data, which is when organizations or governments upload data to the Web in a format enabling it to be re-used and built on. Linked Data is a subset of the wider Semantic Web movement.

See also: The State of Linked Data in 2010

BBC World Cup Website

The biggest sporting event of the year was the soccer World Cup, which was widely covered in the media. The BBC World Cup 2010 website used "dynamic semantic publishing" technology to enhance its daily World Cup reporting.

The site featured over 700 webpages and was powered by a semantic publishing framework. It boasted a comprehensive ontology (a map of concepts), that output "automated metadata-driven web pages" created on-the-fly. It was an impressive demonstration of how a large, mainstream website can add meaning and structure.

There you have it, ReadWriteWeb's selection of the top 10 Semantic Web products and implementations of 2010! Let us know in the comments whether you agree or not with our top 10.

The Mobile Web Today: Location is Just "Tip of the Iceberg"

Berners-Lee began his keynote by discussing the improvements we've seen in technology in recent years, most notably the ability of our devices to be location-aware. However, he says, "location-awareness is just the tip of the iceberg." Devices already know so much about you: your geographical position, which way is up, which direction you're headed, etc., but future devices may know more than this. For example, they may know about your medical information and your physical state. Perhaps they could tell when you're excited by measuring heart rate increases, he said.

Another major improvement which is impacting the Web is the explosion of data available online, a project which he has heavily contributed to, here in the U.K. with data.gov.uk and its U.S. counterpart data.gov. Not so long ago, less than 10 years ago, in fact, "data" on the Web consisted of governments uploading a scanned document, like a spreadsheet that would be posted as a PDF on a government-hosted website. If anyone ever wanted to do anything with that data, they would have to re-type the information. Today, that same type of data is more accessible - the raw data itself is available and, says Berners-Lee, there's a race between governments and other organizations as to who can provide the best and most interesting data.

As for how data access relates to mobile, Berners-Lee explained that data drives development on mobile, just as it does on the Web as a whole. Even a basic calendaring type of application is data-driven. By combining the aspects of mobile technology, like location-awareness, with the semantic Web of data, entirely new types of mobile applications can exist. Most recently, augmented reality applications are an example of this pairing, tying together location with data to identify points of interest just by aiming your mobile phone at something like a landmark or building.

4 Concerns about the Future of Mobile

All that being said, Berners-Lee also mentioned that there are concerns which we need to be aware of when moving forward with mobile technology. They are as follows:

1. Privacy

The challenge of privacy is one many companies, both mobile and otherwise, have been dealing with in recent months. However, on mobile phones, the problem that has not been worked out yet is how to allow a user to share their location while still making it easy for them to understand when they're sharing critical information, how much control they have over that information and who can access that data. The challenge here is how to do all this without getting in the way of the user's experience.

The solution, says Berners-Lee, is that we may need to re-adjust our ideas about privacy. "I think that we'll end up having to think about privacy from a different point of view," he said.

2. Accountability

Along the same lines of user privacy, is the idea that companies that want access to our critical information have a responsibility to build systems that respect that data. "Responsible" companies that are accountable for how they use our data are key. Clearly, this is a struggle many companies are dealing with now, and no one has a winning formula yet.

3. Neutrality

A perennial favorite topic for Berners-Lee is the idea of network neutrality, referring to regulations that forbid prioritizing the speed or access with which one company's data is available over another's. Companies that want you to use their services have an incentive to end neutrality for their own benefits - for example, those that provide voice services may want to slow down access to VoIP services.

Here Berners-Lee was the most passionate, saying point-blank that "the moment you let neutrality go, you lose the Web as it is - you lose the idea that you can click a link and go anywhere."

4. Bringing Web Access to the Rest of the World

The last point also involved a project in which Berners-Lee is involved: providing Web access to the 80% of the world that doesn't go online. He works on this issue through the foundation at webfoundation.org, which examines the challenges in this area. Surprisingly, lack of signal with which to log onto the Web is not the main thing holding back the spread of the Web. 80% of the world has access to the Web, but, for some reason, chooses not to use it.

The cost of data is partially to blame in many cases for this, and for those who cannot afford data plans through their carriers, they're limited to SMS for sharing information. But SMS is very constraining, says Berners-Lee. What's needed instead are better, more low-cost data plans for mobile phones. Carriers should want to offer these plans because once people get a taste of what a data plan can provide, they're potential customers for an upgrade to a more expensive plan that offers even more data and would generate more revenue for carriers.

Affordability of Web access is an area which Nokia thinks about when building their technologies. For example, Nokia's Ovi Maps service uses compression so as not to need data access when zooming in and out, unlike competing service from Google Maps. Nokia's messaging services also compress data and as, Mary McDowell, Nokia's EVP of Mobile Phones, mentioned in the keynote speech following Berners-Lee's, Nokia's recent acquisition of mobile Internet company Novarra was primarily for access to its proxy-browsing browser technology, which saves on clicks, while also providing faster and more efficient access to the Web. This is an important technology for emerging markets where data plans are pricey, but needed.

Disclosure: Nokia paid for this reporter's travel and accommodations to Nokia World 2010.

Invites available! Click through for link.

The engineers freely admit that Meshin's user interface (UI) is currently the Achilles' Heel of the app. It's nowhere near as polished and put together as competitor Xobni's, for example. But they'll fix that, they promise. "We're hiring a UI designer," they tell us.

Focusing on the looks, though, is missing the big picture. Meshin is different from other email-based contact management systems including not only Xobni, but analysis engines like Gist, too. Where those companies hinge on the person - here's their title, where they work, their emails, attachments, their blog posts, their last Twitter update, etc. - Meshin actually analyzes the information found in the information streams it examines. It then extracts related conversations, related messages, related people and other semantically understood data. And it does so by looking beyond keywords. It knows what things mean. It knows if a word is referring to a person, place or thing. It can also surface related links and news from the Web for any given entity.

Already, the engine behind Meshin isn't limited to email messages alone. For example, if you subscribe to RSS feeds within Outlook, those are also understood as being a part of the relationship map with another person. If you subscribe to Twitter feeds within Outlook, again, those are analyzed, along with the other streams.

Meshin arose from a Xerox-funded project inside PARC whose goal is to commercialize older PARC IP for a broader audience. The project has been in development for only a year, with a small core team and support from PARC researchers. The long-term goal for Meshin is to extend itself beyond Outlook, in order to connect other types of information streams together. 

The researchers are contemplating where they should take the technology next - another email platform? An RSS reader? A standalone product? Should they open up Meshin APIs (application programming interfaces) for developers to use within their own applications and services? All these models are a possibility, but first the engineers wanted to just get the technology out there, in the hands of users.

We're helping them with that by distributing invites to the private beta. For access, click here.

]]> http://readwrite.com/2010/09/06/parc_releases_new_semantic_technology_in_form_of_an_outlook_plugin http://readwrite.com/2010/09/06/parc_releases_new_semantic_technology_in_form_of_an_outlook_plugin Product Reviews Mon, 06 Sep 2010 00:45:52 -0700 Sarah Perez SPARQLZ is a stealth technology project aimed to provide a graphical user interface for everyday users to assemble, edit, share and mash-up modular, persistent, real-time searches across the web of Linked Data. It's a side project by an independent team within a large data corporation, with dreams of spinning their work off as a startup.

It's a pretty hot idea: it's like Yahoo Pipes, for Linked Data - but easier to use and already populated with big sets of valuable information to mashup and parse. Linked Data is a growing field of datasets that are categorized with standardized markup, tied together and easily cross referencable by machines. The US and UK governments, news organizations, music data bases, social networks and other organizations are participating in the official W3C Linked Data community. Now SPARQLZ aims to make all that data easy to construct future-facing search queries for.

SPARQLZ is named after the SPARQL query language for structured data. The service also uses technologies like Yahoo Query Language and real-time push format PubSubHubbub. Search results can be delivered to SMS, Email, Webhooks, a Feed Reader or other SPARQLs.

How would I use a service like this? Just imagine stringing if/then statements together through the cloud of Linked Data (see below). As a technology publisher, I'd like to receive notification if and when SPARQLZ finds a photo (from Flickr or CORDIS) of a person from a known tech company (defined as the companies listed in Crunchbase), that's headquartered in a country with a lower than world-median GNP (per the CIA World Fact Book). Could SPARQLZ do that? In theory, it could do things like that all day long - and you and I could trade queries like that back and forth like legos to piece together whatever kind of stream we were looking for. It's a really exciting vision.

Linked Data is growing fast and has incredible potential as a development platform. Sir Tim Berners-Lee, widely credited as the key inventor of the World Wide Web, is now focused on the growth of Linked Data as the web's next step.

But in order to get beyond the borders of Wonky-land, Linked Data needs a good User Interface. For many people, the ability to set up dynamic queries, mix and match them and have them deliver alerts to various devices could scratch an itch that many of us didn't know we had.

Ok, so let's be honest. This is probably not going to be a mainstream phenomenon. But are there thousands, tens of thousands or maybe a larger number of people who could create value for themselves using a query construction and publishing model like this? Who could not or could not so easily create that kind of value on top of Linked Data today? I think there are. Maybe there are even millions of people who could capture some of the latent value in Linked Data thanks to a tool like this.

SPARQLZ is certainly intended as a way to democratize creation and use of something rich with value but previously too technically inaccessible for many people to use who might like to. In that it's of the same vein as Blogger and WordPress were to text publishing, as YouTube is to video publishing and Twitter and Facebook are to social activity feeds.

Might this be the project that makes Linked Data hacking something that far more of us can engage in? That would be great, but first the team will need to gather support and launch itself as a company. For the sake not just of this small team of data-loving dreamers deep inside a big corporation - but for the sake of all us data-loving dreamers who would love to use their tool, I hope they can do it.

The latest version of the Linking Open Data dataset cloud, as at July 2009, maintained by Richard Cyganiak and Anja Jentzsch.

Hunch co-founder and Chief Product Office Caterina Fake told Wired in a recent profile that "the ultimate goal of the company is to map every person on the Internet to every object on the Internet, be that a product, a service, or a person."

I visited the Hunch web site today and answered more than 20 questions, in exchange for which I was offered a list of recommendations of magazines, books and TV shows. It's not a perfect list - I doubt I'll ever watch (connect to, follow) The West Wing, for example, no matter who or what recommends it to me. Nevertheless, Hunch is onto something.

Why Hunch Exists

The so-called Web 2.0 era of the Web was based on user-generated content and social networking around that. Services like YouTube, MySpace and Flickr (which was co-founded by Caterina Fake) were the success stories of that era.

But now, in 2010, there is too much user-generated content to manually process. What's more, social networking is practically dominated by one company: Facebook. We no longer rely so much on niche sites like Flickr, YouTube, Netflix, Amazon to connect to other people socially. Another aspect to consider is that there's a lot of new data streaming in from sensors, RFID tags and other Internet-connected objects.

The upshot is that we need web services that can help us process all of this data and connect us to the parts that are personally relevant to us.

Opportunities For Startups

The refreshing thing is that these trends are opening up huge opportunities for startups.

GetGlue is another example of a startup aiming to match social data to objects or media. It knows for example that I recently watched Inception and (mostly) liked it. GetGlue can use that piece of data about me, look at my history of other movie likes, connect that to the movie history and preferences that it knows about other people who liked Inception. Ultimately all of that social and 'like' data can be used by GetGlue to recommend other movies to me that I may like to see.

We're early in this era, but both Hunch and GetGlue are busy building up extensive databases about people and what they like (their "taste" data). Not only that, they're slowly perfecting recommendation engines that process this data - ultimately filtering, structuring and personalizing it.

Let us know what other 'beyond social' startups have caught your eye recently.

While this is not trivial, I am nonetheless beginning to question the long-term effects of an API-centric worldview, a sort of blind faith in the almighty API, which has at best a difficult relationship with open data and big data concepts.

As CEO, guest author Shion Deysarkar is responsible for the overall business and development of 80legs. In a previous life, he ran a predictive modeling firm. He enjoys playing poker and soccer, but is only good at one of them.

How Do We Access Data Today?

There are two core ways to access data today – via a publisher or via a crawler. Each has a different role.

Publishers have data and choose to make it publicly available through an API so that developers can easily design products powered by a given service.

Crawlers on the other hand are used to proactively go out and grab data by yourself - scraping Web pages for whatever it is you're looking for, data that can then be used to build products and inform better product and marketing decisions.

There is something of a third option, as well: data aggregators like Factual and Infochimps and Hoovers. I'm not going to treat them as part of this post because they gain access to data like the rest of us – via APIs and/or crawlers. They facilitate the distribution of that data as part of their core business (most often using a marketplace concept or subscription), but the input mechanisms are no different.

And there is potentially even a fourth option – human curation of the kind that Factual and WolframAlpha and CrowdFlower employ to acquire new data altogether. But all of these providers offer API access to their data, so I'm still going to bucket them as such.

APIs, at least as we think of them today, have many disadvantages. And before you grab your shovels and organize a mob to come after me, please understand that I'm not calling for the discontinuation of APIs.

At 80legs, we ourselves offer a popular API, which takes a particularly hybrid approach – providing programmatic access to the data acquired via crawls.

What I really want is a natural stratification based on who is good at what, essentially. Right now, we're asking APIs to do too much.

APIs are great for the real-time web, for example – they're great for staying up to speed, whether that means trending search data or retweet velocity. APIs are great for enhancing functionality – whether that's a Klout score or geolocation. APIs are great for integrating certain pieces of non-strategic infrastructure like invite codes (Prefinery) if you're a startup in beta, or Freshbooks, if you're an accountant. They're also great for app-level integrations, like adding Facebook accounts to Tweetdeck, or sucking down content from Netflix.

But at a higher level, as all applications and services become more and more data-driven, it's important to understand the differences between these different methods for extracting data, regardless of where you net out philosophically.

This is a discussion that needs to take place, but too few people recognize the distinctions.

Control, Control, Control

Control and flexibility are the two most important elements to look at when it comes to the difference between an API and a crawl. I also spend some time at the end of this post talking about security and privacy, because I think there are big impacts in these areas for APIs and crawlers alike.

Cost might be a fourth facet to look at, but that's grounds for a different post because pricing varies so widely.

Let's start with control.

When using an API, publishers – companies like Amazon and LinkedIn, for example – control the entire process. Publishers provide you with an API account, which allows you a certain amount of calls, or requests for data per day. They also determine what kinds of content are made available, and in what context.

Publishers offer an API for many reasons. It's financially in their best interest to have products built on top of their data to increase developer loyalty and form a kind of API-dependency to their content. It's also useful as a way to accurately measure server usage and overall engagement, even if there's no money involved.

APIs can go down and become unavailable, they can go from free to paid, and their publishers can be acquired by larger companies that make all manner of changes. There's a lot of uncertainty in APIs, and many devs have learned this fact the hard way. Think back to Gnip rethinking their entire business model due to the relicensing of certain APIs.

But like moths, we so often head right back to the flame.

Crawlers act very differently. They allow much more control over the data acquisition process. This has many advantages.

For starters, the format in which content is delivered can be a lifesaver if formatted properly, or prompt hours of additional work if not. APIs supply content in one format – the format chosen by the publisher. Say you need a XML file type but the company only delivers JSON through their API. You're either stuck or left spending hours re-formatting.

Crawlers let the choice-driven developer have his cake and eat it too. Formats are just another choice to make beforehand, instead of a hindrance.

Granted, standardization can be great in some cases – for example with sites like MySpace where each profile is customized and therefore rendered in HTML differently. MySpace APIs format the content to make it uniform, meaning that what was once difficult to work with as a developer (i.e. large discrepancies in the data), is now standardized and simple to use.

But the "one size fits all" mentality fails more often than you might think, especially once you step outside of the Web's largest sites – one size fits all rarely fits anyone well.

And it's not just format – crawling offers much more control when it comes to time and timing, scope, and cost, too.

Flexibility and Availability

Data access choices are an important component of building any Web product, especially when it comes to flexibility and availability. Specs change, needs change - heck, markets change. Especially if you're a lean startup, out early plus iterate often is a way of life.

APIs only deliver content from the publisher's site. You're locked into a single interface's content sources and structure, without flexibility by definition, which can be very limiting. You're left with acquiring stand-alone datasets to supplement your evolving needs, or mashing up with another API to fill in holes.

Now, the very best API providers are great at adapting to developers' needs and evolving alongside them. Companies like Yolink, for which its API is its bread and butter, are particularly responsive. But too often an API is left unattended, having been a mere box to check, instead of a strategic commitment.

Unimaginative APIs can also limit use cases unwittingly, because some of the furthest-flung (if more promising) applications just aren't supported in the calls or code. There's a huge difference between an API that wants to be heightened and explored and an API whose scope, if anything, constrains original thinking.

Crawlers on the other hand aren't specific to any one site's data, meaning that they can access content from any number of sources and compile it in one place, mixing and matching, comparing and contrasting to your heart's delight.

Crawls can be more open-ended and investigative as well, whereas an API is more about putting a square peg in a square hole. API's also don't offer competitive advantage; everyone has access to the same stuff. A clever crawl can help build a moat.

Finally, crawlers can reach far beyond the capabilities of an API. Millions of pieces of data are publicly available on the Web, and only a very small percentage of it is available via an API. At a certain point it's purely an issue of volume. Much of the Web is instantly crawl-able, and the amount of data available freely on the Web is growing more quickly than the number of APIs by an order of magnitude. The caveat is that you just have to know where to look.

The Elephant in the Room - Security and Privacy

Let's talk about privacy and data, because how the world evolves in this respect could have huge implications for APIs and crawlers alike.

As the recent Facebook data privacy concerns highlight, the security of people's data is a high priority, regardless of how it may or may not be acquired or sold. Further, users expect publishers to protect their data aggressively (whether they do is another matter).

This is a PR and perception issue as much as it is anything else.

Users worry that their data might get into the hands of people who will use it for malicious purposes, whether via an API or a crawler. I would argue that this is not always the case because responsible crawling companies, at least, have strict licensing agreements with their clients to ensure data is used lawfully.

But the reality is that publishers are increasingly incentivized because of public policy issues to constrain API access. And the world's biggest crawler, Google, is starting to look evil, with the ominous question, "What exactly does Google know about me?" popping up at family dinners around the country.

Some are even arguing that Facebook is bound to be federally regulated sooner or later because of its profligacy when it comes to data, and that would certainly have broad impacts.

APIs are not inherently more or less secure than crawlers, but in the current climate, especially with regards to privacy, we can expect companies large and small to make less and less data open and available (something that the linked data community has been ruing as well).

Security right now is a big X factor that is going to take some time to play out.

The nice thing about crawlers (depending on your perspective) is that they are harder to control, at least for now. But it is a reasonable thing to say that data responsibility and privacy issues are going to shape and reshape this conversation big time.

Today's Web is full of data that if kept within an API-driven paradigm suffers from less creative use, less flexibility, and less control (from a developer standpoint).

An endlessly crawl-able Web was in many ways what Tim Berners-Lee and WC3 intended for the Web all along. Content creators like publishers and social networks can create sites as they've always done, while data aggregators can access data in whatever format they like.

In fact, in an older but still applicable interview with Berners-Lee, he talks about why an open, linked data Web is by far preferable than APIs for data access.

There is a foundational, DNA-level need to share data. Without openness, you lose the full value and impede any future innovation in the process. APIs absolutely have benefits – but only when we are not beholden to them – when we can use them rationally, strategically, and carefully. And when data isn't at the crux of your site, service or application.

"We have an open API" is an overused phrase, especially as APIs are by no definition open or closed.

If you need certain attributes, like real-time or speed, certain capabilities, or certain pieces of infrastructure, there are thousands of amazing APIs out there. But if your business runs on data, crawling is the only way to go.

Top photo by Jason V. Tim Berners-Lee photo by the Knight Foundation

It would be easy to say that the 'social' element is still the primary part of today's Web, since the popular products of this era enable you to say what's on your mind (Facebook), what's happening (Twitter), or where you are (Foursquare). All of these are mostly social activities. But more significantly, these and other products output data that will increasingly be used to build personalized services for you.

The more data there is, the better Web services will be at delivering personal value to you. While part of this increase in data is coming from social data from the likes of Facebook and Twitter, much of it is coming from the Internet of Things and data uploaded by governments and organizations. In short: the read/write Web is now much more than the Social Web.

How We Went Beyond Social

So how did we arrive at a Web that is less about social and more about you?

After the peak of Web 2.0, we (meaning all of us) began to get overwhelmed with the choice of content available. We thought we had to actually 'read' as much of that content as possible. So we watched YouTube, chatted on MySpace and Facebook, read blogs, followed lots of people on this new thing called Twitter, and so on. By the end of 2008, we were exhausted by all of this CONTENT. How could we possibly keep up?!

In 2010, we're still struggling to digest all of what social media throws at us. However, a shift has been happening since 2009 which alleviates the problem. We've begun to realize that it's not how much content we consume that is important: it's what we do with all of the social and other data available to us. The social is still important, but the resulting data is - slowly - becoming more important because it can be analyzed, filtered, mashed up and personalized.

Structured Data & Internet of Things

Two relatively new trends are driving this change.

The first is the increasing amount of data being uploaded to the Web by governments, organizations and people. Much of this data is being structured using Semantic Web technologies like RDFa or microformats. In other words, it is categorized and encoded with meaning that machines can process. Recent examples include U.S. and U.K. government data, Best Buy's store and product data and Facebook's Open Graph.

And then we have the Internet of Things: an evolving trend where real-world objects and 'things' are connected to the Internet via technologies such as sensors and RFID tags - everything from cars to houses to roads and more. The upshot is that the Web is about to experience a data explosion, as billions of sensors and other data input and output devices upload exabytes of new data to the Web.

How do We Use This Data?

If we add together social data from the likes of Facebook and Twitter, data from governments and businesses, and data from sensors and RFID, this is a huge amount of data. Most of it isn't for "consuming." Rather, the value of all of this new Web data will be in how it's filtered, mixed together ("mashed up") and personalized in new Web services - most of which haven't yet been built.

Adam Greenfield is one of the leading thinkers of the Internet of Things; I interviewed him earlier this year about his book called Everyware. Greenfield recently wrote a post describing a near future scenario for non-technical people using the Web. He posited a use case where his mother would be able to plan a train trip to see her son, by creating an "ad-hoc service" that tapped into the Web and utilized real-time data sources.

In 2010, his mother would have to find and "read" several different applications in order to plot her travel schedule, and some of that information isn't even currently on the Web. Greenfield envisions a near future where his mother can essentially "write" her requirements into her mobile or other device, and the Web will deliver a personalized schedule to "read." You can view a diagram of Adam's concept here (PDF).

Don't Think Social, Think Data

Successful products in the Web 2.0 era had a strong social element: YouTube, MySpace and Flickr were a few relatively early examples. In the current era of the Web, which began to form in early 2009, the focus has shifted from social to data-driven software. Successful products of this era of the Web will be ones that filter, structure and personalize this vast amount of data coming onto the Web.

So if I was an entrepreneur or developer wondering what to build for this era of the Web, I wouldn't be thinking social. I'd be thinking: How can I use all of this data and build on top of it? There are incredible opportunities out there for you.

This current era of the Web doesn't have a name, which is probably a good sign! One thing is for sure though: It's still a read/write Web - only now you're reading and writing data from much more than just social services. You're increasingly interacting with "things," organizations, governments - virtually anything that can connect to the Web.

It's an impressive demonstration of how a large, mainstream website can have added meaning and structure.

The BBC World Cup site featured over 700 webpages and was powered by a semantic publishing framework. The site boasted a comprehensive ontology (a map of concepts), that output "automated metadata-driven web pages" created on-the-fly.

Jem Rayfield, Senior Technical Architect, BBC News and Knowledge, explained further:

"The underlying publishing framework does not author content directly; rather it publishes data about the content - metadata. The published metadata describes the world cup content at a fairly low-level of granularity, providing rich content relationships and semantic navigation. By querying this published metadata we are able to create dynamic page aggregations for teams, groups and players."

The basis of this system was an ontology that described how World Cup facts related to each other. For example, "Frank Lampard" was part of the "England Squad" and the "England Squad" competed in "Group C" of the "FIFA World Cup 2010". The ontology also included "journalist-authored assets" such as stories, blogs, profiles, images, video and statistics.

The publishing platform had both manual and automated tagging features. BBC journalists could, for example, tag Frank Lampard in a story about the disallowed goal from England's last-16 loss against Germany. This is a normal part of most modern-day publishing systems (we tag content in this manner here at ReadWriteWeb). But the BBC World Cup site also automatically analyzed journalist content and matched it "against the World Cup ontology." It did this by using what it describes as a "natural language and ontological determiner process." [Update: IBM wrote in to inform us that the technology behind this was IBM LanguageWare.] This is similar to software such as Thomson Reuters' Calais or the new Extractiv product that we reviewed yesterday. In the BBC's case, the resulting tags were moderated before being published.

The BBC used Semantic Web technologies such as RDF and SPARQL to build their World Cup site. The stated goal was to achieve "intelligent mapping of journalist assets to concepts and queries."

The site reportedly served millions of page requests a day throughout the World Cup. The BBC may use this semantic publishing platform for other parts of the BBC sports site; and it will certainly deploy it again for the the London 2012 Olympics.

The official explanatory post has the technical details, should you wish to follow up. Let us know in the comments about other large scale Semantic Web deployments that you know of.

Update: See also The World Cup and a call to action around Linked Data, by the BBC's John O'Donovan. Thanks Georgi Kobilarov for the pointer.

I sat down with Extractiv CEO Shion Deysarkar at the recent Semantic Technology conference in San Francisco, to find out how Extractiv intends to compete with the more well-known and big media backed Calais.

How Extractiv Works

Extractiv is a joint venture between Houston-based web crawling service 80legs and natural language processing company LCC (which created Swingly, a Q&A service).

Deysarkar explained that Extractiv uses technology from both of its parent companies, to crawl the Web for content on a particular topic and then - using natural language processing - transform it into structured data. This video, produced by Extractiv, explains how the service might be used to crawl the Web for stories about smart phones over the past month.

The output of the crawl and analysis can be JSON or XML, two formats commonly used for structured data. Support for RDFa, a popular Semantic Web standard, will be available "soon" according to the company. Extractive also offers an API, allowing customers to bypass the web site.

Extractiv is free to try, but if you'll be a moderate or heavy user of the service then you'll have to pay (the pricing is as yet unavailable on the web site).

Extractiv vs Calais

Deysarkar told ReadWriteWeb that Extractiv is targeting "mid-market Calais customers" - such as media companies or those developing search applications, monitoring services, recommendation engines or aggregators. He also claimed that Extractiv goes beyond what Calais offers, because it can mine sentiment data (which is data about how people feel about products and services).

Extractiv also wants to "provide access to more types of semantic information than any other provider." As CEO of partner company LCC, Andrew Hickl, put it, "if you're interested in baseball pitchers, a generic type like PERSON just won't cut it."

At launch, Extractiv offers about 250 different types of named entities, but it aims to have more than 3000 different entity types by the end of the U.S. summer.

Preparing For the Future of the Web

The product is not aimed at the consumer market, so it's not for the faint hearted and you need to know what to do with all of that XML or JSON data! It also remains to be seen how competitive it is with Calais, which is a proven performer and has many reputable companies as its customers. Some startups have taken on Calais before, but fallen short.

However, there is undoubtedly a need for products like Extractiv and Calais that turn the Web's unstructured data into meaningful, organized content. This is the future of the Web, because there is going to be a large increase in the quantity of data online over the next 5-10 years - and all of that data will need to be structured if we're going to be make the best use of it.

Mobile and Real-Time Web have been particularly eventful in 2010, as you'll see below. Augmented Reality and Internet of Things are both early stage trends, but have continued to edge towards the mainstream this year. The movement towards Structured Data has made significant progress in 2010, primarily thanks to RDFa and the adoption of that Semantic Web format by Facebook, Google and other big companies.

Mobile

The next generation of the iPhone operating system, iPhone OS 4, was launched in April - it included long-awaited support for multitasking. That was followed by the June release of iPhone 4, the new handset.

However, it hasn't all been about the iPhone this year. Perhaps even more notable (although less hyped) has been the continued growth and expansion of Google's Android competitor. It started in January, when Google announced the new Nexus One. Google described it as "Where Web Meets Phone" and called it a "super phone" (ok, so there was hype there too...). Android, Google's mobile OS, has also experienced strong market share growth this year.

The biggest news of the year, though, has undoubtedly been Apple announcing the iPad in late January. This led to a flurry of new apps and a revival of interest in digital magazines.

In terms of non-device innovation, much of the attention has been on location-based social networking services like Foursquare, Gowalla and Brightkite.

While all three were popular during the annual geekfest in Austin, SXSW Interactive, by the half-year mark Foursquare appears to have the most momentum.

Real-Time Web

Twitter and Facebook have dominated the Real-Time Web landscape so far in 2010. Facebook has been under pressure for its controversial privacy changes (that is, more and more of Facebook is being made public), yet it continues to grow market share. Meanwhile Twitter has become more well-known in the mainstream, on the back of news such as Michael Jackson's passing and huge events like the World Cup.

Google tried to get in on the real-time action in February, with the launch of a media sharing service called Google Buzz. Buzz was mostly a damp squib, however. More successful for Google was its integration of Twitter, MySpace, Facebook and other real-time results into its core search product.

There have been some interesting startups to emerge in this space over 2010. For example, Quora - a real-time enabled Q&A site created by Facebook's first CTO Adam D'Angelo. Other products to have caught our eye this year include Collecta, OneRiot, My6Sense and SuperFeedr.

Internet of Things

Internet of Things is when real-world objects are connected to the Internet, often via sensors, barcodes and RFID tags. This was a trend we began exploring in earnest in 2009, when much of the applications were experimental.

In 2010 we've seen commercial applications begin to arise, starting at the 2010 Consumer Electronics Show (CES) in January. The main trend we noticed from this CES was web applications being ported to consumer electronics - everything from the technology inside cars to Web-enabled TVs.

Many of our posts on Internet of Things this year have been explanatory, to help you prepare for the expected increase in commercial applications over the next few years. Check out our post on sensors and mobile phones for an example. Also find out why HP thinks sensors will lead to the next big wave of computing.

Augmented Reality

The field of AR has been similarly experimental up till now, so we have again spent a lot of time explaining this trend and putting it in context. An example is Austrian company Wikitude bringing augmented "Worlds" to the iPhone, in February. A wider audience saw AR in April, when the Discovery Channel promoted its docu-drama hit Deadliest Catch with a desktop-based AR ad campaign.

Augmented Reality was a hot topic at the RWW Mobile Summit, in May in Mountain View, CA. Photo: Chris Cameron

Structured Data

If 2009 was the year of "open data" (when previously offline data is uploaded to the Web), then 2010 so far has been the year of RDFa. RDFa is a lightweight way to add extra meaning to HTML web pages. Facebook is using it in their Open Graph platform, which was announced in April. The stated goal of the Open Graph protocol is to enable publishers to "integrate [their] Web pages into the social graph."

Others that have climbed on board the structured data train include Google and retailers like Best Buy and Tesco. Twitter hasn't yet supported RDFa, but its Twitter Annotations project comes close.

The open data movement also continues to expand. In January, Data.gov.uk, a new web site dedicated to making non-personal data held by the U.K. government available for software developers, launched with the help of Sir Tim Berners-Lee.

You can see that 2010 has been very active with innovation! The iPad, new mobile devices, the increasing market penetration of Facebook and Twitter, the rise of the Semantic Web - it's been a fascinating year so far! In the comments, let us know your personal highlight of 2010 so far.

Lead image: Steve Rhodes

At the recent Semantic Technology conference, I sat down with Evri CEO Will Hunsinger. He called Evri the "topical equivalent of a Facebook stream."

The technology of Evri is much the same as it was two years ago - it connects together topics using Semantic Web algorithms - but the landscape in which Evri is playing has evolved. In 2010, real-time information streams dominate. So Evri now aims to be a curation service.

In a nutshell, Evri allows you to explore and track topics. Its homepage displays a current hot topic (at time of writing, it was Lance Armstrong and the Tour de France), with other trending topics offered in the menu (for example 'Gulf Oil Spill' is a featured trend and 'LeBron James' is listed as "Trending in US & World'). You may also enter your own topic into the search bar. Evri is like a mix between Google Trends and Google News, with liberal sprinklings of Twitter and Facebook.

Whither Twine...

Evri made the news earlier this year when it acquired failed semantic web bookmarking application, Twine. Curiously, Hunsinger described Evri as "the inverse of Twine" - because Evri does all the work, rather than the user. Twine is an application that relies on its users to actively bookmark pages, a la the much more successful Delicious. Evri automatically collates topical information and presents it the user.

Update: Twine founder Nova Spivack wrote in and stated: "Twine was in fact highly automated as well, but in a different way than Evri - Twine used NLP [Natural language processing] to auto-tag every entry, generate summaries, and used graph algorithms to make recommendations."

Where Evri shares similarities with Twine is in the ability for users to track a topic. Hunsinger said that Evri users may "follow a story as it evolves over time, and tune it." He described this as being like a "mini blog" for its users.

Twine appears to have been of most use to Evri for its underlying technology. Hunsinger told us that Evri is using technology it acquired from Twine to extend Evri's categories and for advanced filtering.

What's Next

Evri is not short on ideas and innovation - for example it announced a Sentiment API last August. However some of these ideas are slow to eventuate. Hunsinger said that the Sentiment API is not in commercial deployment yet, because it requires much more media in order to calculate sentiment and it tends to work best with well-known people (like Barack Obama). However, Evri is working on incorporating data about shares, tweets, and more in order to beef up its Sentiment analysis engine.

The company is also currently working on what Hunsinger described as "Pandora-like recommendations," referring to the geographically-limited online music service Pandora.

Other expansion plans include launching one new channel per week, to extend Evri's topical coverage.

Can Evri Compete as a Consumer App?

Evri is an interesting product and is currently being used by media partners like Hearst and Canwest.

However, Evri will likely continue to struggle as a consumer offering. It's competing against a plethora of real-time news apps - everything from TweetDeck, Google News, Topix, Techmeme, and more. I wouldn't be surprised if Evri is eventually acquired by a big media company for its technology, much as Evri snapped up the struggling Twine for the same reason.

There is perhaps no better example than Facebook's use of RDFa. We chatted to Facebook open standards evangelist David Recordon to find out more.

In April this year, Facebook announced a large-scale new platform called the Open Graph. The stated goal of the Open Graph protocol was to enable publishers to "integrate [their] Web pages into the social graph." Essentially, each web page can now become an 'object' in Facebook's social graph (which is Facebook's term for how people connect to each other in its network). This means that pages can be referenced and connected across social network user profiles, blog posts, search results, Facebook's News Feed, and more.

The Open Graph in a Nutshell

The Open Graph is a wide-ranging platform which includes features such as 'Like' buttons and publisher plug-ins. It also includes a simple, RDF-based markup. This requires publishers to include at least 4 metadata properties in each object: title, type, image, URL. There are a few additional properties which may be optionally added, such as site_name and description.

When the Open Graph was announced, ReadWriteWeb feature writer Alex Iskold wrote a comprehensive guide to the Open Graph for publishers, users and competitors. Here's how he described the markup:

"...the markup enables publishers to say what object is on the page - a movie, a book, a recording artist, an event, a sports team, etc. This automatically enables semantics, that is, an understanding that the user is not just interacting with a web page, but that he or she is liking a specific kind of thing. Semantics then leads to bucketing of the objects into categories like books, movies, music, etc., and gives rise to all sort of applications, including personalized recommendations. Perhaps even more importantly, the markup helps Facebook connect the users across common interests across different web sites."

The syntax that Facebook uses is fairly similar to RDFa, without being an exact match. Sandro Hawke from the W3C (the Web's official standards body), told ReadWriteWeb last week that Facebook is using RDFa "in an abbreviated, not really good modeling way."

Facebook's KISS (Keep it Simple Semantics)

The reason that Facebook is using an abbreviated version of RDFa is that they want to make it as simple as possible for publishers to deploy it. And the W3C is fine with this, because it has a rules standard called RIF that can convert Open Graph code into proper RDFa if needed. Also the W3C and Semantic Web community seem to be having fruitful discussions with Facebook.

We asked Facebook open standards evangelist David Recordon why the company chose not to use exact RDFa syntax. He replied that simplicity was paramount, so they wanted to use as few properties as possible.

However, Recordon said that Facebook's code will actually help the adoption of RDFa. He thinks that RDFa will become a more interesting technology when more people are consuming it (via Facebook's variation of RDFa).

Facebook did not set out to support the Semantic Web, added Recordon, it just wanted simple code that was as close as possible to open standards. He told us that Facebook found the "sweet spot" between the complexity of existing Semantic Web standards and the simplicity required for tens of thousands of developers to place 'objects' inside their web pages.

Open Graph Issues

Facebook's Open Graph protocol has its issues, which Alex Iskold outlined in a follow-up analysis in May (it should be noted that Iskold's startup, GetGlue, has been mining web page metadata for a while now - so it's partly competitive with Facebook, although it also provides support for the Open Graph). Some of Facebook's issues were due to poor implementation of the markup by Facebook partners, but Recordon told us that much of this has been fixed now.

Perhaps a bigger problem is that currently only one object can be specified per web page, which prevents multiple topics or people on a single web page from being semantically marked up. Also, Iskold pointed out that there is no way to disambiguate objects - so if two objects have the same name, they are considered the same thing by Facebook. For example, 'Paris' could refer to the city in France or the city in the USA (also it could be the name of an infamous socialite).

Kick Starting the Semantic Web

These issues ideally should be ironed out, but for now it's clear that Facebook wants Open Graph to be very simple so that as many publishers as possible implement it. That's a sensible business decision, although it means that we're only getting a limited version of the Semantic Web for now. However, the support of Semantic technologies by big companies like Facebook, Google and Best Buy will be enough to kick start this new era of a smarter and more structured Web of data.

What do you think, is Facebook doing enough to support Semantic Web standards? Let us know your thoughts in the comments.

This is not just an academic exercise for Best Buy. As we will see, semantic technology has already led to increased traffic and better service to its customers. We spoke to Jay Myers, Lead Web Development Engineer at BestBuy.com, to find out how.

Myers told us that the primary goal of using semantic technologies was to increase the visibility of its products and services. And with data such as store name, address, store hours and GEO data being marked up using RDFa, search engines are now able to identify each of those data components more easily and put them into context.

A quick refresher on the terminology: just as the lingua franca of the Web is HTML (Hypertext Markup Language), RDF (Resource Description Framework) is commonly thought of as the primary language of the Semantic Web. RDFa is a kind of 'lite' version of RDF, which adds metadata to HTML (or XHTML) webpages.

The process of adding RDFa to Best Buy's webpages began two years ago, when the company began to look for ways to get more visibility to its stores on the Web. "At that time," said Myers, "it was difficult for users to find basic store information like store location and hours."

To solve this dilemma, Best Buy gave each store its own blog.

Best Buy employees entered information into the blogs every day, using online forms that output RDFa. Myers told us that the use of RDFa makes "human input from our store employees more visible on the Web."

Best Buy is using Good Relations, a Semantic Web vocabulary for e-commerce that describes product, price, and company data.

Myers remarked that "there isn't a noticeable difference" to the users of Best Buy's website, however all of the RDFa data is very visible to humans via search engine results and its store locator tool. The RDFa data is "also great for machines," said Myers, which has resulted in "a definite up tick in the amount of search traffic to these pages." At last week's SemTech conference, Myers said that it had resulted in a 30% increase in search traffic. He noted that Best Buy hadn't expected to see an SEO benefit, but it's been a boon to them since the company is "very reliant on search engines" for product discovery and store locations.

With Jay Myers at the development wheel, Best Buy's web presence will continue to be enhanced by the Semantic Web. RDFa can ultimately create rich relationships between products, which will in turn "create a deeper visibility to additional products" when a customer is shopping.

That seems like a distinct competitive advantage for Best Buy.