Home 80Legs: A Web Crawler as a Service

80Legs: A Web Crawler as a Service

80Legs is a web crawling and online content analysis service which first impressed us back in April at the Web 2.0 Expo. At that time, the company was launching into a private beta, but today at the DEMOfall 09 conference, they’re going live. In the time that has passed since their initial debut, the company has been working on scaling out the performance and power of their service while also preparing to launch a new feature which should appeal to both developers and non-developers alike: an “app store.” This feature allows 80Legs users to write applications that run on top of the 80Legs service and gives them the ability to share those apps with others.

With 80Legs, anyone can have their very own search engine to command and control, and now thanks to the apps, they can have it do anything they like with just a click of a button.

What 80Legs does is no easy feat. It provides its users a service which offers up 50,000 computers which can crawl up to 2 billion web pages per day. Yes, it’s like having your own little search engine that you can rent for a small fee. How small? 80Legs is about 50% less expensive than any other competitive service out there.

While consumers may not have a use for a service such as this, there is an extensive market that does. 80Legs aims to attract customers from a wide variety of disciplines including alternate search engine developers, market researchers, IP protection services (those who go after copyright infringements, article theft, etc.), competitive intelligent services, and ad networks looking to audit their own ads and see where other ad networks are placing theirs.

80Legs App Store

In addition to the big news about the service’s public launch, 80Legs is also revealing their development program. With this new feature, developers can write their own applications that run on top of the 80Legs service and then make them available to others through a soon-to-launch “app store.” Here, other customers can browse and purchase apps that suit their needs whether it’s something for media analysis, market research, sentiment analysis, or whatever else the developer comes up with. The developers get to set their own CPR price for the apps and get to keep 100% of the revenue earned, too.

The API for app building was actually made available to beta users a couple of months prior to DEMO, but the company plans to soon support multiple languages which will include Java, .NET, Perl, and Python, so developers can work in whichever they feel most comfortable with.

While there are other ways to crawl the web, 80Legs wants to make sure that there’s nowhere else you can do it for such a small fee. If you’re interested in trying 80Legs for yourself, you can do so as of today by signing up at 80Legs.com. Use the code “RWW” to receive an additional 50% credit on top of the amount you put in. (First 50 users only).

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.