Why Pinterest Needs To Upgrade Visual Search Stat

Pinterest isn’t a social network. It’s a visual search engine. And its latest purchase of Visual Graph further proves its growing self-awareness of this fact.

Millions use Pinterest to curate appealing images by designating them to specific categories called “pinboards.” But beneath the surface, our interaction with the service is not unlike an enormous visual search engine indexed by people instead of machines. It’s a ton of visual data, and Pinterest has only begun the process of figuring out how to surface it. 

Now, Pinterest is prioritizing a way to more easily manage this data—by hiring the two man team behind Visual Graph. Both ex-Googlers, their image-recognition and indexing technology is what makes Google’s reverse image search tick. 

Why Text Search Doesn’t Work For Pinterest

As a major player on the Visual Web, Pinterest couldn’t help but make enormous strides in visual search over the years, simply to remain functional.

Modern search engines have been developed to identify textual keywords, so the concept of search by image is a major paradigm shift. Even Google relies on adjacent text content in order to identify images (and reverse image search just matches like pixels). 

For awhile, Pinterest’s internal search engine was dependent on textual cues like alt text, words in the image link, and the user’s image description, to identify photos. However, this led to problems with spammers for Pinterest circa 2011 through 2012. If a user wanted to make any photo show up as the top search result for “cat,” they’d just have to write “cat cat cat” in the description to inadvertently—and sometimes intentionally—cheat the system. 

Not to mention, words with double meanings could be deceiving. While “food porn” has always been a popular topic on Pinterest, it’s against the rules to post actual porn. But for a time, Pinterest couldn’t tell the difference. In those days, it was up to a six-person anti-spam team in the engineering department to remove porn almost manually. 

When I talked to former engineering lead Jon Jenkins about image search last fall, he said Pinterest was beginning to read data directly off of images themselves. One example: Pinterest can now automatically determine and deliver an image’s dominant color while it loads. 

“We do feature analysis, which cuts an image into boxes and compares it. We can determine if it’s a closeup of another image on the site,” he said. “The more data you have, the easier it gets.”

Dominant colors and image matching are cute gimmicks, but they're no revolution in image identification. Acquiring the technology to go further is the next logical step. 

Visual Search For The Visual Web

Visual Graph founder Kevin Jing and colleague David Liu will be heading Pinterest’s new “visual discovery team,” according to a Pinterest spokesperson.

Output from Visual Graph's object detectors. Output from Visual Graph's object detectors.

Based on what Visual Graph has already accomplished, we can deduce that Pinterest is going to take its image-search technology light years forward with this acquisition. 

According to Jing and Liu, Visual Graph combines big data elements with detailed individual image analysis, or as their latest post says, “Our approach is to combine the state-of-the-art machine vision tools, such as object recognition (e.g. shoes, faces), with large-scale distributed search and machine learning infrastructures.”

The company has developed technology to identify objects depicted in images, including skirts, purses, and cars. Face recognition at Visual Graph, notes Jing, is on par with that at Facebook and Google. The end goal is to organize these images into graphs categorized by the objects they include. If this feature were applied to Pinterest, users could unearth pictures of clothing even if the original pinner didn’t post a description or a relevant link title. 

There’s no doubt Pinterest’s investors haven’t yet honed in on the benefits of a search engine that can identify pictures of cars, clothes, and handbags. If it became easier to search Pinterest by commodity, it could become more of a shopping destination than it already is, and perhaps even make good on its $3.8 billion valuation

Pinterest is already the Visual Web’s most notable search engine—just not a very good one. So far, it relies too much on textual and user created context. But this latest acquisition indicates Pinterest’s eagerness to change that. 

Image by mkhmarketing