By Alex Iskold
In October last year, Netflix launched an unusual contest. The online movie rental company
is offering 1 million dollars to anyone who can improve their recommendation engine by
10%. Netflix is known for its innovation and
bold moves and in the grand scheme of things, $1M is not a lot of money for such a
business.
The competition is still running (it “continues through at least October 2, 2011”), so
is this a publicity trick or an attempt to do research on the cheap? Is better recommendations
something that Netflix really needs or is it just nice to have? Today Netflix is facing a
challenge from the awakened giant BlockBuster, so it is certainly looking for a
competitive edge. A great recommendation system can retain and attract users to the
service. For example when a user returns a movie, he/she is recommended another movie
they might like – which increases the likelihood of return business.
Browsing and Recommendations
A good recommendation engine can make a difference not just for Netflix, but for any
online business. This is because there are two fundamental activities online –
Search and Browse. When a consumer knows exactly what she is looking
for, she searches for it. But when she is not looking for anything specific, she browses.
It is the browsing that holds the golden opportunity for a recommendation system, because
the user is not focused on finding a specific thing – she is open to suggestions.
During browsing, the user’s attention (and their money) is up for grabs. By showing
the user something compelling, a web site maximizes the likelihood of a transaction. So
if a web site can increase the chances of giving users good recommendations, it makes
more money. Obviously this is a difficult problem, but the incentive to solve it is very
big. The main approaches fall into the following categories:
- Personalized recommendation – recommend things based on the individual’s past
behavior - Social recommendation – recommend things based on the past behavior of similar
users - Item recommendation – recommend things based on the thing itself
- A combination of the three approaches above
We will now explore these different approaches by looking at old-timers like Amazon
and newbies like Pandora and del.icio.us.
Amazon – The King of Recommendations
Amazon is considered a leader in online shopping and particularly recommendations.
Over the last decade the company has invested a lot of money and brain power into
building a set of smart recommendations that tap into your browsing history, past
purchases and purchases of other shoppers – all to make sure that you buy things. Lets
take a look at various pieces of Amazon’s recommendation system to get an insight on how
they work. Here are the sections that are shown in the main area of my Amazon account
when I login:
The section above shows Social recommendations. Notice that it is
very analytical, giving me a statistical reason for why I should buy this item. Also note
that this recommendation is also a Personalized recommendation, since it
is based on an item that I clicked recently.
The section above shows Item recommendation based on New Releases.
Clicking on the Why is this recommended for you? link takes me to a view of my
purchasing history. So this recommendation is also a Personalized
recommendation, since it is based on my past behavior.
There are four more sections offered on the page and each of them leverages different
combinations of the personalization mechanisms described above. We summarize them in the
table below:
Not surprisingly, the system is symmetric and comprehensive. All recommendations are
based on individual behavior, plus either the item itself or behavior of other people on
Amazon. Whether you like to buy something because it is related to something that you
purchased before, or because it is popular with other users, the system drives you to add
the item to the shopping cart.
Beyond Amazon
The Amazon system is phenomenal. It is a genius of collaborative shopping and
automation that might not be possible to replicate. This system took a decade for Amazon
to build and perfect. It relies on a massive database of items and collective behavior
that also “remembers” what you’ve done years and minutes ago. How can new companies
compete with that?
Surprisingly, there is a way. The answer is found in a subject that has little to do
with online shopping – genetics. As you know, this science studies how pieces of
DNA, called genes, encode human traits and behavior. For example, members of a family
look and behave alike because they share a certain subset of genes. Genetics as a science
has been around for over 150 years and has been a powerful tool for both medicine and
history. But on January 6, 2000 things took an unexpected turn – Tim Westergren and his
friends decided to apply the concepts of genetics to music.
Pandora – The Recommendation System Based on Genetics
The Music Genome Project was launched
to decompose music into its basic genetic ingredients. The idea behind it is that we like
music because of its attributes – and so why not design a music recommendation system
that leverages the similarities between pieces of music. This kind of recommendation
engine falls into the Item recommendation category. But what is new and
profound here is that similarity of an item like a piece of music needs to be measured in
terms of its “genetic” make up.
After years of struggle and processing massive amounts of music, the project
accumulated enough data and launched the service called Pandora. Pandora became a hit because of its precision
and low cost of entry. The user just needs to pick one artist, or a song, to create a
station that instantly plays similar music.
This kind of instant gratification is difficult to resist. The fact that Pandora
understands what makes music similar allows it to hook the user without having to learn
what this user likes. Pandora does need the user’s tastes or memory, it has its own –
based on music DNA. Sure, sometimes it might not be perfect, as the user’s taste might
not be perfectly addressed. But it is rarely wrong.
The natural question is can this genes-based approach be applied to other areas – like
books, movies, wines, restaurants or travel destinations? What constitutes genes for each
category? For example, can we say that for wine, the genes might be things that describe
how wine tastes: blackberry, earthy, fruity, complex, blend, etc. And for a book, can the
genes be phrases that describe the plot? So if the genes are the attributes of the object
that make it unique in our mind, we should have no problem coming up with genes for
various things. In the past few years we have been doing this a lot online. It’s called
tagging!
Pandora had a big startup cost, because thousands of pieces of music had to be
manually annotated. The social bookmarking phenomenon del.icio.us took a different approach – let people annotate
things themselves. This self-organizing approach has worked really well, and del.icio.us
quickly became popular among early adopters. Today, del.icio.us is considered to be more
than bookmarking destination – it is also a news site and a search engine. But is
del.icio.us a recommendation system?
The answer is yes. There is a basic recommendation system based on one gene – a single
tag. For example, in the picture above we see popular links for the linux tag
and we also see related tags like open source and ubuntu. But a much
more exciting recommendation system is based on matching multiple tags. Unfortunately,
the current heuristic does not always work, which is why it is not obvious. But luckily,
it did work for the Read/WriteWeb page and generated a great list of similar blogs (see
“related items” below):
So the del.icio.us approach holds intriguing possibilities of self-organizing
classification and recommendation systems. With enough users and more tweaking, social
tagging can result in a system that works equally well for books, wine and music.
Provided, of course, that tags are so good that they become genes!
Conclusion
Recommendation engines are important pieces of online commerce systems and their user
experience. Retailers have a big incentive to provide recommendations to those users who
are “just browsing”, to drive them towards a transaction. Amazon.com, the leader in the
space, has a very compelling personalization offering. The problem that other retailers
face is lack of user information and infrastructure.
Recent approaches to recommendation engines, like the genetics-inspired Pandora and
social tagging pioneered by del.icio.us, are intriguing. These approaches hold the
promise to provide instant gratification, without asking the user to reveal her
preferences and past history. Regardless of how things unfold in the future, Amazon,
Pandora and del.icio.us are examples of extraordinary recommendation technologies. We
commend them and are watching in fascination for what is coming next.