Home The Art, Science and Business of Recommendation Engines

The Art, Science and Business of Recommendation Engines

By Alex Iskold

In October last year, Netflix launched an unusual contest. The online movie rental company
is offering 1 million dollars to anyone who can improve their recommendation engine by
10%. Netflix is known for its innovation and
bold moves
and in the grand scheme of things, $1M is not a lot of money for such a
business.

The competition is still running (it “continues through at least October 2, 2011”), so
is this a publicity trick or an attempt to do research on the cheap? Is better recommendations
something that Netflix really needs or is it just nice to have? Today Netflix is facing a
challenge from the awakened giant BlockBuster, so it is certainly looking for a
competitive edge. A great recommendation system can retain and attract users to the
service. For example when a user returns a movie, he/she is recommended another movie
they might like – which increases the likelihood of return business.

Browsing and Recommendations

A good recommendation engine can make a difference not just for Netflix, but for any
online business. This is because there are two fundamental activities online –
Search and Browse. When a consumer knows exactly what she is looking
for, she searches for it. But when she is not looking for anything specific, she browses.
It is the browsing that holds the golden opportunity for a recommendation system, because
the user is not focused on finding a specific thing – she is open to suggestions.

During browsing, the user’s attention (and their money) is up for grabs. By showing
the user something compelling, a web site maximizes the likelihood of a transaction. So
if a web site can increase the chances of giving users good recommendations, it makes
more money. Obviously this is a difficult problem, but the incentive to solve it is very
big. The main approaches fall into the following categories:

  • Personalized recommendation – recommend things based on the individual’s past
    behavior
  • Social recommendation – recommend things based on the past behavior of similar
    users
  • Item recommendation – recommend things based on the thing itself
  • A combination of the three approaches above

We will now explore these different approaches by looking at old-timers like Amazon
and newbies like Pandora and del.icio.us.

Amazon – The King of Recommendations

Amazon is considered a leader in online shopping and particularly recommendations.
Over the last decade the company has invested a lot of money and brain power into
building a set of smart recommendations that tap into your browsing history, past
purchases and purchases of other shoppers – all to make sure that you buy things. Lets
take a look at various pieces of Amazon’s recommendation system to get an insight on how
they work. Here are the sections that are shown in the main area of my Amazon account
when I login:

The section above shows Social recommendations. Notice that it is
very analytical, giving me a statistical reason for why I should buy this item. Also note
that this recommendation is also a Personalized recommendation, since it
is based on an item that I clicked recently.

The section above shows Item recommendation based on New Releases.
Clicking on the Why is this recommended for you? link takes me to a view of my
purchasing history. So this recommendation is also a Personalized
recommendation
, since it is based on my past behavior.

There are four more sections offered on the page and each of them leverages different
combinations of the personalization mechanisms described above. We summarize them in the
table below:

Not surprisingly, the system is symmetric and comprehensive. All recommendations are
based on individual behavior, plus either the item itself or behavior of other people on
Amazon. Whether you like to buy something because it is related to something that you
purchased before, or because it is popular with other users, the system drives you to add
the item to the shopping cart.

Beyond Amazon

The Amazon system is phenomenal. It is a genius of collaborative shopping and
automation that might not be possible to replicate. This system took a decade for Amazon
to build and perfect. It relies on a massive database of items and collective behavior
that also “remembers” what you’ve done years and minutes ago. How can new companies
compete with that?

Surprisingly, there is a way. The answer is found in a subject that has little to do
with online shopping – genetics. As you know, this science studies how pieces of
DNA, called genes, encode human traits and behavior. For example, members of a family
look and behave alike because they share a certain subset of genes. Genetics as a science
has been around for over 150 years and has been a powerful tool for both medicine and
history. But on January 6, 2000 things took an unexpected turn – Tim Westergren and his
friends decided to apply the concepts of genetics to music.

Pandora – The Recommendation System Based on Genetics

The Music Genome Project was launched
to decompose music into its basic genetic ingredients. The idea behind it is that we like
music because of its attributes – and so why not design a music recommendation system
that leverages the similarities between pieces of music. This kind of recommendation
engine falls into the Item recommendation category. But what is new and
profound here is that similarity of an item like a piece of music needs to be measured in
terms of its “genetic” make up.

After years of struggle and processing massive amounts of music, the project
accumulated enough data and launched the service called Pandora. Pandora became a hit because of its precision
and low cost of entry. The user just needs to pick one artist, or a song, to create a
station that instantly plays similar music.

This kind of instant gratification is difficult to resist. The fact that Pandora
understands what makes music similar allows it to hook the user without having to learn
what this user likes. Pandora does need the user’s tastes or memory, it has its own –
based on music DNA. Sure, sometimes it might not be perfect, as the user’s taste might
not be perfectly addressed. But it is rarely wrong.

The natural question is can this genes-based approach be applied to other areas – like
books, movies, wines, restaurants or travel destinations? What constitutes genes for each
category? For example, can we say that for wine, the genes might be things that describe
how wine tastes: blackberry, earthy, fruity, complex, blend, etc. And for a book, can the
genes be phrases that describe the plot? So if the genes are the attributes of the object
that make it unique in our mind, we should have no problem coming up with genes for
various things. In the past few years we have been doing this a lot online. It’s called
tagging!

Del.icio.us – Can Tags Become Genes?

Pandora had a big startup cost, because thousands of pieces of music had to be
manually annotated. The social bookmarking phenomenon del.icio.us took a different approach – let people annotate
things themselves. This self-organizing approach has worked really well, and del.icio.us
quickly became popular among early adopters. Today, del.icio.us is considered to be more
than bookmarking destination – it is also a news site and a search engine. But is
del.icio.us a recommendation system?

The answer is yes. There is a basic recommendation system based on one gene – a single
tag. For example, in the picture above we see popular links for the linux tag
and we also see related tags like open source and ubuntu. But a much
more exciting recommendation system is based on matching multiple tags. Unfortunately,
the current heuristic does not always work, which is why it is not obvious. But luckily,
it did work for the Read/WriteWeb page and generated a great list of similar blogs (see
“related items” below):

So the del.icio.us approach holds intriguing possibilities of self-organizing
classification and recommendation systems. With enough users and more tweaking, social
tagging can result in a system that works equally well for books, wine and music.
Provided, of course, that tags are so good that they become genes!

Conclusion

Recommendation engines are important pieces of online commerce systems and their user
experience. Retailers have a big incentive to provide recommendations to those users who
are “just browsing”, to drive them towards a transaction. Amazon.com, the leader in the
space, has a very compelling personalization offering. The problem that other retailers
face is lack of user information and infrastructure.

Recent approaches to recommendation engines, like the genetics-inspired Pandora and
social tagging pioneered by del.icio.us, are intriguing. These approaches hold the
promise to provide instant gratification, without asking the user to reveal her
preferences and past history. Regardless of how things unfold in the future, Amazon,
Pandora and del.icio.us are examples of extraordinary recommendation technologies. We
commend them and are watching in fascination for what is coming next.

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.