news.internacionalesnoticias.netCorrespondents · Reports · Analysis
CORRESPONDENT REPORTTechnology

AI Crawlers Strain Library Websites and Limit Public Access to Collections

Published
Nov 05, 2026
Desk
Technology
Views
883

Automated scraping is overwhelming library websites, disrupting access to free academic, historical and government collections for everyday users.

AI Crawlers Strain Library Websites and Limit Public Access to Collections

AI crawlers are putting open library collections under strain

Libraries are discovering the cost of making information freely available: automated scrapers can overwhelm the same public websites that students, researchers and ordinary readers rely on. Some collections have been knocked offline as AI firms and other companies gather material for large language models. At San Jose State University, Jill Strykowski, who leads cataloguing and government documents, says bots are bombarding the institution’s online resources to collect as much material as possible. Libraries, museums and other cultural institutions are attractive targets because their collections often contain vetted academic, historical and government records without paywalls or login requirements. The disruption has been reported across North America, Europe and Oceania since late 2024, according to [researchers studying the problem](https://www.glamelab.org/products/are-ai-bots-knocking-cultural-heritage-offline/). Experts say it has continued to worsen. That’s more than a technical nuisance: when bot traffic takes a library site down, public access disappears along with it.

When scraping crowds out readers

Dean Seeman, director of collections and digital infrastructure at University of Victoria Libraries, says heavy bot traffic has sometimes brought websites down temporarily. Peter Lewis, Canada Research Chair in Trustworthy Artificial Intelligence at Ontario Tech University, describes the result as ordinary users competing with effectively unlimited automated traffic. Lewis also sees a larger imbalance: public-interest institutions make their collections available for the common good, while private companies can use that material to build commercial products and sell them back to the public. The data doesn’t fully explain how much of the traffic comes from AI developers themselves versus third-party services, but the pressure on library systems is already clear.
Two people, seen from above, work side by side on laptops. There are papers strewn nearby.
Workers enter data from an old document belonging to the McMillan Memorial Library during a digitization exercise at the Eastland Library in Nairobi on March 24, 2022. Libraries and other cultural heritage institutions are prime targets for AI bots. (Patrick Meinhardt/AFP/Getty Images)

Good bots and bad bots

Eric Hellman, executive director of Project Gutenberg, says the distinction is not simply between automated access and human access. Search engines such as Google and archive services such as the Internet Archive use crawlers that follow robots.txt, a site’s voluntary instructions about what automated visitors may do. Some AI scraping bots ignore those instructions. They can request thousands of documents per second and download the same material in multiple formats, putting a site under strain. Hellman, whose U.S.-based volunteer organization digitizes and distributes cultural works as ebooks, called poorly programmed bots “monstrosities” in a [blog post last year](https://go-to-hellman.blogspot.com/2025/03/ai-bots-are-destroying-open-access.html). He says the problem has worsened since then. The scale can be startling. The University of North Carolina at Chapel Hill reported receiving more than 11,000 searches for Finnish music in a single day, from thousands of internet addresses. Bot traffic periodically forced its library catalogue to block students, faculty and staff. Blocking individual IP addresses is increasingly ineffective, Hellman says, because one company or bot operation can distribute requests across thousands of addresses.
A man poses in front of a colourful wall.
Michael Weinberg, co-director of GLAM-E, a collaborative effort focused on open-access policies for cultural institutions, is working to protect public access to information. (Submitted by Michael Weinberg)
Project Gutenberg is willing to let AI companies train on its material, Hellman says. Most of the collection is public domain, and the organization even offers a one-click download of its books. His complaint is that the bots fail to use that option responsibly and instead risk taking websites offline.

Protecting access without shutting the door

Libraries have tried CAPTCHA tests and other checks to screen out automated traffic. Michael Weinberg, co-director of GLAM-E, an [academic initiative](https://www.glamelab.org/products/are-ai-bots-knocking-cultural-heritage-offline/) at NYU Law School and the University of Warwick focused on open-access policies, warns that those measures can also block legitimate crawlers. Some institutions have begun requiring users to log in. The experts who spoke with CBC News say they’d prefer to avoid that approach because it puts new barriers between the public and collections meant to be readily available. And as scraping tools improve, they may find ways around today’s checks. AI companies have broadly argued that scraping publicly available material is allowed under fair use and serves the public interest. For libraries, the immediate question is more practical: how to keep collections open when automated visitors can overwhelm the systems that provide access.
A man is seen, backlit, against a large sign reading 'Anthropic'
The Anthropic logo is displayed during the Dreamforce technology summit in San Francisco, on Sept. 17. AI firms have broadly argued that scraping publicly available data is permissible under fair use and in the public's best interest. (Carlos Barria/Reuters)

A voluntary signal has limits

After facing public criticism and copyright lawsuits, Anthropic, OpenAI, Google and Microsoft said last week they supported a shared opt-out signal in a site’s robots.txt file. The signal would tell their crawlers not to collect that site’s data. That’s a start, but the commitment is voluntary. It also doesn’t cover troublesome bots operated by third parties or smaller AI companies. For libraries and cultural institutions, the gap matters: a rule followed only by some crawlers can’t guarantee that putting collections online is safe.

Digitization has opened valuable resources to far more people. Weinberg called that access “fantastic,” pointing out that 20 years ago, reaching some materials might have meant driving or flying, or securing special permission.

“I think it’s been a huge benefit to society that libraries and museums and other cultural institutions have put a huge amount of work into making their collections available,” Weinberg said. But today’s bot traffic can discourage institutions from publishing parts of their collections. Regular visitors to library websites are, one expert says, competing with effectively limitless traffic from bot farms.

Several people sit at long tables, reading books and laptops, in an enormous high-ceilinged library.

People look at their laptops and in the main reading room of the New York Public Library on Oct. 5, 2016, in New York City. Increased bot traffic means regular people using library websites are effectively competing with infinite traffic from bot farms, says one expert. (Dominick Reuter/AFP/Getty Images)

Groups such as GLAM-E are working on lasting ways to preserve online access for people and helpful bots. The test will be whether those protections can keep pace with automated systems that ignore voluntary signals. If you work with public collections, that’s the unresolved question: how to keep access open without making the institutions that provide it pay the price.

Source: Kevin Maimann · www.cbc.ca

Discussion

Sign in to join the discussion.