But was this a fair fight?
Mobile analyst firm MSearchGroove has just published the results of a series of tests which show that the mobile search service ChaCha beat out two other voice-enabled search applications on the iPhone when it comes to search query accuracy. [Update, Ed: a commenter points out that the report was actually sponsored by ChaCha] To test this, the researchers used Google’s own mobile application and Vlingo for iPhone, an app that lets you search both Google or Yahoo. Oddly, they ignored Yahoo’s mobile app, which also has voice search built in.
The results of their study aren’t entirely shocking: if you want to be understood, ask a human, not a computer.
The Mobile Search Tests
ChaCha’s mobile search service can be accessed both by SMS and by calling a toll-free 1-800 number. Since these tests focused on voice search, the phone-in method was used. When using ChaCha, the service identified the queries accurately in 94.4% of the cases and delivered accurate search results 88.9% of the time. Vlingo, which the researchers used to test Yahoo search, only interpreted queries correctly in 72.2% of the cases and delivered accurate results 27.8% of the time. Google, surprisingly, fared worst of all. Their mobile application only understood spoken queries in 16.7% of tests and delivered accurate results 22.2% of the time.
To test the applications, the researchers conducted two rounds of tests using both keyword search and natural language queries where they asked questions using sentences. The queries represented a cross-section of typical mobile searches in categories like navigation, directions, local search, general information, social search, and long-tail search.
It’s not all that surprising to find that ChaCha outperformed the other voice-enabled applications – after all, they have real, live humans on the other end of the line to interpret the spoken questions. What is surprising, though, is how wide the gap is in between the human-powered search and the speech recognition apps, especially when contrasting ChaCha with Google.
Did Google Just Get Beaten at Search?
When you think of search, you tend to think “Google.” When you use Google, there’s a certain expectation that your queries will be interpreted accurately and your results will be relevant. What these tests show, however, is that when it comes to the mobile platform, all bets are off. Not only was Google outperformed by a mobile application whose name few mainstream users have probably heard of (Vlingo), they were also outperformed by a crowdsourced workforce who answer ChaCha queries in their spare time. Could this mean that mobile search is an area – perhaps the only area – where a competitor could actually get a foothold and steal away a bit of Google’s market share?
Well, not so fast. Google could still dominate on mobile thanks to brand recognition alone. Mainstream users aren’t going to seek out new alternatives to search, even if they’re better. That’s precisely why companies like Microsoft have to spend millions of dollars on advertising campaigns just to gain a percentage point or two of search market share.
Then there’s the fact that tests which compare human-interpreted queries to machine-interpreted ones seem a little unfair. We all know that people can still understand each other much better than computers can. (Well, for now at least). ChaCha wins this round, but only because this was never a fair fight to begin with. Speech recognition and natural language processing are technologies still in their infancy. But if we know Google, they’re coding away right now to improve them as we speak.
