Written by Alex Iskold and edited by Richard MacManus

In an article in the January 1st 2007 issue of NYTimes, reporter Miguel Helft writes about the race in Silicon Valley to beat Google. Certainly the future of search has been much talked about lately. Last year Read/WriteWeb had a number of big posts on this topic, including Emre's Search 2.0 and my post about vertical search. We have also profiled many search players, including Retrevo, Hakia, Quintura, Pluggd, Microsoft Live Search, Snap and ChaCha. Since we have been following the battle closely, we are excited to see the coverage in New York Times - which signals that the search space has heated up enough to be worthy of attention by a tech-savvy mainstream audience.

The reporter Miguel Helft works through the main issues of the race. He explains that a huge amount of venture money has been poured into the search space - not necessarily to beat Google, but perhaps to get little slices of the gargantuan advertising market. However, Helft concludes that even though many companies are trying to catch up, he says it's not clear whether they can. Google is not a monopoly like Microsoft, he states, but still it seems unfeasible to unseat to Google at this point. 

Obviously, we all have the same feeling about Google's dominant position - but why? To understand, we need to segment Google's competitors and their approaches. In this post we will use Emre's categorization to see why we all intuitively feel that Google is going to remain the king of search.

The categorization of competitive technologies

There are two classical business cases for a new company: a better way or a cheaper way. In the case of the search market, cheaper simply does not apply. So the companies in the current search space are racing towards different shades of 'better'. In his post, Emre outlined three major approaches to outdoing Google:

• Better Technology - the results are of better quality/relevancy
• Better UI - the results are presented in a better way
• Vertical/Semantic Search - a combination of better technology and better UI

Better Technology

Google's automated algorithm, based on the original PageRank idea, is a phenomenon. The contenders to overthrow Google are deploying two radically different strategies: 1) natural language processing/AI techniques and 2) power of people.

In the AI camp, Hakia and Powerset are building technologies that aim to deliver better search results by better understanding what you mean. Both companies have received a lot of attention in the press lately and we also reviewed Hakia here recently. While it is clear that there are benefits in understanding natural language, what is not clear is how much difference it makes in the quality of the results.

Based on what we have seen so far, it is difficult to see how these companies can beat Google. Firstly, being able to enter the query using natural language is already allowed by Google, so this is not a competitive difference. It must then be the actual results that are vastly better. Now that is really difficult to imagine. Somewhat better maybe, but vastly different? Unlikely.

People powered search is interesting, since it brings to the game what computers lack the most - common sense. del.icio.us is not typically thought of as a search engine, yet it is probably the best example of people power in action. The great thing about del.icio.us is that we get high-quality filtered results. The not-so-great thing is that the results are limited to what most people have found popular. Arguably this is the intent, but in some cases it will also be the limitation. 

Another problem is that del.icio.us has only been used so far by techies, so the results in non-technical areas (medicine for example) are not going to be great. A different approach is taken by ChaCha, which literally has people help you find things. It is interesting idea, however in our review in December we pointed out a few issues - including high monthly burn rate (since search guides have to be paid) as well as potential scalability problems.

While it is clear that solutions like del.icio.us are superior to static web directories, it seems that the search algorithm is going to be difficult to beat by power of people. The speed and completeness of the Google algorithm is going to be particularly out of reach when it comes to people powered solutions.

Better UI

Google's motto is simplicity. This idea runs through their entire product stack and is closely guarded by the company. This is why many competitors are racing to present their search results in a better way.

Snap and Microsoft Image Search are showing Previews to improve the user experience. Previews are interesting and useful. I particularly like the Microsoft Image Search UI, which is a big improvement in how image search is presented. Unfortunately both of these technologies have to improve on relevancy and even quantity of results. Also what works well for images, does not work as well for search results - since it is difficult to determine if it's worth drilling in, purely based on the preview.

Next we look at Clustering solutions, that essentially attempt to do what vertical search engines do - but across the board and using the entire result set. Clusty is very impressive, because it manages to add a lot of value without changing how we think about search results. Along with the search results, the tab with clusters allows the user to refine the query - to basically get to the answer in no more than 2 steps. 

The second engine in this category is SearchMash, which is a Google property and apparently their playground. What is striking about SearchMash is that its ascetic look - so typical of Google - is actually creating a richer way of viewing the search results. It's done using Ajax and has a mode that displays nothing but the headlines of the results. I found this mode to be surprisingly useful. There are also knobs for slicing the results along verticals like Blogs, News and Images - which are very straightforward. Both of these sites (Clusty and SearchMash) seem very useful and capable of reaching mainstream acceptance.

In the last sub-category of better UI, we find Quintura and Kartoo applying innovative Visualization technologies to search. Quintura deploys a graph/tag-cloud like approach to present search result clusters. In a way, the idea is very similar to Clusty - just done in a different way. Kartoo deploys a graph/heatmap-like approach to do exactly the same cluster partitioning. The problem with both of these approaches is that they are too complex, in my view, to make it to mainstream. Even though I have a background in graph visualizations, I do not find either one obvious or intuitive. It is unlikely that in this incarnation, visual search would be widely accessible.

A lot of these UI improvements seem neat and interesting, but none of them is going to be able to beat Google. The main reason is that it is straightforward to replicate them. Of course legally there can't be exact clones, but in this case close enough will also be good enough. None of these technologies has barrier to entries for Google, so if any of them gets enough market share Google is simply going to roll out an upgrade. Note that their toying at SearchMesh.com can already hurt some current vertical search engine providers.

Vertical Search

There has been a lot written about vertical search and for a good reason. These technologies aim to unseat Google by having both better technology and better UI and focusing on a specific vertical. There are a lot of players going after Blogs, Classifieds, Electronics, Health, Jobs and Travel engines - because of both popularity and monetization opportunities.

Consider this simple example. If you wanted to find blog posts about web 2.0, would you use plain Google or Google Blog search? Clearly even Google itself recognizes the utility of having specialized, focused search - if not for every vertical, at least for some like Blogs and Music. So there is no doubt that vertical search makes sense and it is here to stay. The key question is how much of the Search market can vertical search engines bite off?

Actually, it could be a lot. The categories that we named are all money generating/short-pass to transaction kind of categories, which are highly sought after by advertisers. So what exactly does that mean? Given that Google already has specialized search for Blogs and Music, it seems like it is going to be just a matter of time before Google ventures into other categories. Depending on how soon this happens, it might mean a lot of trouble for current players (think Google Blog Search surpassing Technorati). Only those who have managed to build strong brand awareness (like Technorati) will survive.

Conclusion

So overall, even though there is a lot of activity in the space, it seems like Google will remain the search king for the foreseeable future. Various approaches will have different degrees of success in seizing bits of the market, but to make a serious dent will require time, flawless execution, big marketing dollars and, of course, a better technology. This is not a trivial combination of things.

Given that the first three are difficult points, but doable, we still need to understand what better technology means. In my mind it is the one that delivers relevant results faster. From that point of view, another promising contender (mentioned by Emre) is personalized search. With this technology, search results are going to be organized not by PageRank but by your personal interests. It is likely that a combination of a vertical search and personalized search is going to deliver fundamentally better results than Google, so that might have a chance. However, as we pointed out with the other technologies. Google is not going to sleep through this.

Let us know what you think and tell us what alternative search engine is your favorite. Also vote for the search technology you think stands the best chance of beating Google, in our poll: