Home AI scrapers running out of space as restrictions close the net

AI scrapers running out of space as restrictions close the net

TL:DR

  • AI scrapers face increasing restrictions as websites react to data harvesting practices.
  • A study found that 5-7% of 14,000 domains now enforce restrictions, up from 1% last year.
  • Top sites have altered terms of service to prohibit crawling for generative AI purposes.

AI scrapers are increasingly facing hostile online environments as data sources dry up.

Crawling for data, also known as scraping, previously meant vast troves of text, images, and videos could be pulled from the internet without too much trouble. AI models could be trained on the seemingly infinite source but that is no longer the case.

A study from AI research thinktank Data Provenance Initiative, named “Consent In Crisis” has found a hostile environment now awaits website scrapers, especially those for the development of generative AI.

Researchers probed the domains utilized in three of the most important datasets used for training AI models and that data is now more restricted than ever.

14,000 web domains were assessed with the discovery of an “emerging crisis in consent” as online publishers have reacted to the presence of crawlers and the harvest of data. The researchers outlined in the three data sets – known as C4, RefinedWeb, and Dolman – that around 5% of all data, and 25% of content from the best sources had enforced restrictions.

In particular, OpenAI’s GPTBot and Google-Extended crawlers provoked a reaction from websites to change their robot.txt restrictions. The study found between 20 and 33 percent of the top web domains have introduced extensive restrictions on scrapers, compared to a much lesser figure at the start of last year.

Hard crawls resulting in full bans

Over the whole base of domains, 5-7% have enforced restrictions, up from just 1% across the same period.

It was noted that many websites had changed their terms of service to completely prohibit crawling and lifting content for use in generative AI, but not to the extent of the restrictions on robot.txt.

AI companies have possibly wasted time and resources due to excessive crawling that was likely not required. The researchers showed that while around 40% of the top sites used across the three datasets were related to news, over 30% of ChatGPT inquiries were for creative writing, compared to just 1% that featured news.

Other notable requests included translation, coding help, and sexual roleplay.

Image credit: Via Ideogram

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Graeme Hanna
Tech Journalist

Graeme Hanna is a full-time, freelance writer with significant experience in online news as well as content writing. Since January 2021, he has contributed as a football and news writer for several mainstream UK titles including The Glasgow Times, Rangers Review, Manchester Evening News, MyLondon, Give Me Sport, and the Belfast News Letter. Graeme has worked across several briefs including news and feature writing in addition to other significant work experience in professional services. Now a contributing news writer at ReadWrite.com, he is involved with pitching relevant content for publication as well as writing engaging tech news stories.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.