Apple co-founder Steve Wozniak is joining the advisory board of the research engine DeepDyve, a search engine designed to scour the “deep web.” This “deep web” is an area of the internet that isn’t currently indexed by modern-day search engines like Google and yet consists of an estimated 99.8% of the Internet. Any company that is able to successfully tap into this data will be the one to introduce the next breakthrough technology in search as we know it. Will that be DeepDyve?
Wozniak Joins DeepDyve
According to Wozniak, most of the information on the web is “collecting dust because nobody’s come up with a way to mine the data in a way that’s useful to researchers and consumers.” He believes that DeepDyve has the potential to transform Deep Web search and says he’s “excited to bring about that transformation.”
Wozniak’s role at DeepDyve isn’t limiting in any way, but the company expects that his contributions will focus on the DeepDvye technology, especially as it relates to the user experience. As a member of the Advisory Board, he will also meet formally with the company twice per year.
DeepDyve is Not Google
DeepDyve is currently known for their KeyPhrase technology which lets you type in anything in their search box from a few words to entire paragraphs that you copy and paste. The search engine’s algorithm itself was developed by two scientists who worked on the Human Genome Project. As with that project, which required using pattern-matching techniques across large amounts of data, there’s also a need for search engines that can analyze large amounts of data in the same way. That’s precisely what DeepDyve does.
Typically, keyword search on other engines breaks down as queries grow in length – but not on DeepDyve. The more search terms you enter, the more relevant your results. The DeepDyve engine actually encourages longer search queries. This type of search technique certainly comes at a good time, as our query length is growing each year, with 8-plus keyword searches having increased 20% year-over-year as of February, 2009.
Progress and Problems
We looked at DeepDyve back in September when it was still behind a paywall, then reviewed it again in November when they introduced a free version. At the time, we noted that there were still some issues with any “deep web” search engine – most notably that a lot of the information which DeepDyve uncovers is still behind additional paywalls on subscription-based web sites. Today, that issue still remains.
However, the complaints of many of the commenters on the last post were not about the paywalls but about how the site forced you to register before you could do any searches. As one anonymous commenter noted, to paraphrase, “if you’re going to launch a search engine, open it up so people can use it.”
It seems DeepDyve took that advice. The search box on the homepage is immediately accessible and even the results pages have a more refined look today than they did only a few months prior.
These sorts of complaints highlight the problem with reviewing cutting edge technologies when they’re still in such a raw format – people go there expecting Google, find what appears to be a boring research experiment and then become disillusioned. What they fail to see – what they cannot see, in fact – is that a cursory glance can’t reveal the technology behind the site or service. The technology in DeepDyve, for instance, involves advanced algorithms that other engines don’t even have yet. The usability issues will be addressed in time and the issues with access to content behind paywalls could always eventually be worked out through partnership deals. But that business side of the DeepDyve project isn’t anywhere nearly as interesting as the potential of gaining access to 99.8% more of the Internet!
