Home Head to Head Comparison of Text Extraction Algorithms

Head to Head Comparison of Text Extraction Algorithms

A few months ago we linked to Tomaž Kova?i?’s overview of text extraction algorithms. Now Kova?i? has posted an evaluation of several text extraction algorithms and services, including Boilerpipe, NCleaner, the Python and Node.js versions of Readability and the Extractiv API.

To conduct his evaluations, Kova?i? used the cleaneval dataset, which includes 681 documents, and a Google News dataset with 621 documents harvested by the authors of Boilerpipe.


Metric for the Google News data set

A few notes:

  • NCleaner did better on its own Cleaneval data set than it did on the Google News data set, but Boilerpipe did well on both sets.
  • Kova?i?’ was surprised by Readability’s poor performance, and notes the discrepancy between the two ports. He thinks the original JavaScript version may do better.
  • The commercial APIs had the most consistent results.

Image by Andrew Mason

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.