The New York Times and CNN blocked access to content for OpenAI's web crawler GPTBot

News outlets like the New York Times, CNN, Reuters and the Australian Broadcasting Corporation (ABC) have blocked a tool from OpenAI that collects content from their sites.

Here's What We Know

The Verge was the first to report the blocking of GPTBot. Subsequently, The Guardian found that other major news sites including CNN, Reuters, Chicago Tribune ABC and others have also banned the web crawler.

The GPTBot blocking is visible in publishers' robots.txt files, which tell search engines and other organisations which pages they are allowed to visit.

All of the listed publishers added the block in August. CNN confirmed the GPTBot blocking. A Reuters spokesperson said the company regularly reviews robots.txt and the site's terms of service.

The New York Times' terms of service were also recently updated. Specifically, the rules prohibit scraping content for AI training and development.

Flashback

OpenAI is the creator of one of the best-known artificial intelligence chatbots, ChatGPT. Its web crawler, known as GPTBot, can crawl web pages to help improve AI.

Large language models like ChatGPT require huge amounts of information to train their systems. However, developers are often silent about the presence of copyrighted material in their datasets.

To address potential infringements, OpenAI has published information about GPTBot and outlined how websites can prevent the crawler from collecting information from sites whose owners don't want their content used to train AI.

Source: The Guardian