My first post for ReadWriteWeb, just over 1 year ago, started with the premise that search was “game over”, that Google had won and the only space left was (re)search - what users do after the basic search.

None of the search start-ups since then has made me change my mind. None of the cool new user interface features or ways of expressing your search intentions matter one iota, if the core search proposition is not better from day one. Well, enter the latest contender: Blekko.

When Google launched, 10 years ago in 1998, there was no “new paradigm” or wizzy features - just a search box that worked better than the competition. The search competition bar is now way, way higher than it was back then.

Yet new search start-ups continue to get funded, even in what is a less frothy funding environment. Cuil(l) raised $33m. Looks like they blew it. In contrast, Blekko raised only $5m in two rounds. It is still in stealth mode and one assumes they'll will play the hype game a bit more cautiously after the Cuil debacle.

The proposition that launched countless search start-ups was “if we can get just 1% of the search market we will have a very valuable business“. That maybe true, but getting 1% has proved elusive. The reality is you either win big or fail totally in this game. There are no hedged positions in search. It is a really “non-trivial” technical problem.

Assuming the game is defined as the search infrastructure game. I think that game has been over for some time. The barriers to entry are just too high. An entrepreneur pitching VC now has to answer the “how do you avoid the Cuil problem?” Yahoo BOSS is the perfect play in the new game, with search infrastructure players offering their platform to developers. Hundreds of start-ups can make a decent business within less than 1% of the search market if the infrastructure is provided by somebody else. You don’t build operating systems, do you? You don’t build search infrastructure, do you?

You don’t, unless you believe that you really have disruptive technology. Blekko is one the few remaining plays building search infrastructure. They must think they have that disruptive technology.

Blekko seem to understand the complexity of the challenge, from comments on their Blog (as their site says nothing, their founder’s Blog is best source of insight into what they are up to):

“Search is an absolutely fascinating problem to work on for a bunch of reasons. For one thing you have to scale the thing before getting the first user. You can’t just start with a server or two and add more when the users come. Step 1 is to copy the internet onto your cluster. Step 2 is to analyze it..

The componentry is remarkably deep.

Search is like 7 hard problems wrapped into a stack. Distributed systems, html analytics, text analytics/semantics, anti-spam, AI/ML, frontend/UI. And scale…”

His Blog is well worth a thorough read if you are in the search game or just like hard technical problems. (As a historical footnote, Skrenta’s notes on Cuil, written well before the launch, make interesting reading).

Later on he says:

“you don’t need a million servers and half of the phd’s in the field to build a search app. It takes 20 people and $5M of hardware…if you know what you’re doing.”

I totally buy the “It takes 20 people” people bit. All my experience in software has confirmed that Frederick Brooks was totally right in the Mythical Man Month that small teams always outperform large teams. I cannot imagine what more than 20 people would do other than get in each other’s way.

Its the “you don’t need a million servers” bit that I am less certain about. Google invests $ billions in server farms. You have to have something fundamentally and totally disruptive. P2P enabled Skype to take on AT&T and Verizon. That was fundamentally and totally disruptive technology that enables such a compelling value proposition that they got millions of consumers using them. That is why I was excited to see Faroo attempt this with P2P, but I can see that they fail at the critical “has to be better than Google the day it launches” test.

Purely incremental improvements to the economics of crawling + indexing will not enable a new consumer search play. Saying “we only need $1 billion in infrastructure cost to compete out of the gate with Google and Google spent $3 billion” does not cut it with investors. Nobody will fund that $1 billion. However, incremental improvement is a great pitch to the big infrastructure players. If you can say “I can take 20% out of your infrastructure costs with my patented technology”, you will get your phone calls returned by Google, Yahoo and Microsoft. And one of them may offer to buy you for a big fat premium to prevent their rivals getting access to the same technology.

That is very, very different from launching a new consumer search engine.

In summary, I see 3 possible search plays in search today:

1. Build search applications on top of Yahoo BOSS or equivalent offerings from Google or Microsoft. There is room for hundreds of niche, vertical start-ups, using search as a feature not as the only proposition. I think Yahoo has a great shot at this as Google will suffer from cannibalization fears, so they won’t open up as much as Yahoo. Microsoft will undoubtedly play here as well, they are best at technology for developers.

2. Hard core search infrastructure technology sold to Google, Yahoo and Microsoft. That’s tough to get right as the technology has to be really, really good, the patent has to be rock solid and you have be good at playing poker with the big guys.

3. The totally disruptive Skype style venture that nobody has heard about.