According to a post on the Technorati blog today, the original blog search engine is in the process of replacing its old crawler with a new design in an attempt to track and index the blogosphere more efficiently.
While the company has been working on this new infrastructure for over six months and the larger blog platforms have been using the new crawler for some time, Technorati expects to see all blogs utilizing the new crawler by the end of next week.
“Over the last six months we’ve been working on a rewrite of our blog crawling infrastructure from the ground up.” Dorion Carroll writes in the post. “With a new architectural approach, we’ve added a lot better coverage of many new blogging conventions, better visibility and tracking of what is happening in the internals of the crawler, and how to do a better job of recognizing all the great content bloggers are discussing and linking to.”
Over the past year Technorati has been getting a lot of attention for its inability to crawl and index blogs efficiently, so this may be great news for bloggers who have for a long time bemoaned having to manually ping Technorati each time they publish a new post.
And it’s not only the smaller blogs that have experienced this. Only yesterday, while looking at the top 100 blogs in Technorati, we noticed that ReadWriteWeb had not been crawled in 23 hours. That’s a long time considering we had published ten posts and therefore pinged the service ten times in that period of time.
While we can’t be certain that this new crawler will make indexing faster, we’re pleased to see Technorati striving to become more relevant and useful at a time when really great, really fast blog search is essential.
