TechnologyReview has published an interview with Google’s Director of Research, Peter Norvig, where it’s acknowledged that the search giant hires individuals to look at search results pages and provide their judgment about the quality of the results.
This one part of the interview lead to a thoroughly researched blog post by Saul Hansell at the New York Times, one at watch-dog blog Google Blogoscoped and will probably be written about in many more places before the day is through (heck, including here). I can’t stop wondering, though – is this really news or at all a surprise? It’s both fascinating and mundane at the same time.
Here’s what Norvig actually said:
Another way we do it is to randomly select specific queries and hire people to say how good our results are. These are just contractors that we hire who give their judgment. We train them on how to identify spam and other bad sites, and then we record their judgments and track against that. It’s more of a gold standard because it’s someone giving a real opinion, but of course, since there’s a human in the loop, we can’t afford to do as much of it.
This is a matter of great importance, as Google holds so much power over our collective thinking and knowledge these days.
Does this mean that these peoples’ judgment is weighed heavily in determining the results? Are their thoughts used to improve the algorithm or are they acted upon directly? Does the introduction of a human element smack of some unfairness that an algorithm, though created by humans as well, lacks?
It’s not very clear at all just what Norvig means by his statement. For all its prestige and access, the publication that did the interview appears to have done so by emailing a predetermined list of questions to the subject – at least there was no follow up on this very interesting response. As such, there’s any number of ways readers could take it. As one commenter on the NYT says, “Theres an important difference between judging your algorithms results versus incorporating ratings into your algorithm.” The next commenter, though, says that the first is missing the point. “everyone knew that Google was continuously assessing their algorithm, the fact that Google is blending a algorithmic (i.e., determined by math) AND a directory (i.e., determined by people) model is new.” I’m not sure which of them is reading the interview most accurately.
On one hand Norvig’s admission is deeply thought provoking – but on another hand isn’t it obvious that this would be happening on some level. Just like YouTube hires people to watch videos all day and look for copyrighted content, despite all the big promises of a technology to catch the stuff, it would be insane if Google didn’t hire people to look at search results and provide feedback.
I’d love to know what kinds of thoughts this provokes in our readers’ minds. See also SearchEngineLand coverage, where this story is put into historical perspective.
