Merge branch 'kn/clang-format-updates': The Struggle Against AI Scrapers
As you're reading this, it's likely because the administrator of this website has taken steps to protect its server against the increasing threat of AI-powered scrapers. This measure, dubbed Anubis, aims to shield the site from aggressive AI companies that seek to exploit its resources without permission.
Anubis is a compromise solution that uses a Proof-of-Work scheme, similar to Hashcash, to deter mass scraper attacks. At an individual scale, this added load may be negligible, but when applied at a massive level, it becomes prohibitively expensive for scrapers to operate. This hack serves as a placeholder step towards more effective solutions, such as fingerprinting and identifying headless browsers.
However, Anubis comes with its own set of requirements. To bypass the challenge, users must enable JavaScript on their devices. Unfortunately, this means that plugins like JShelter will need to be disabled for this domain. The reason behind this requirement is a shift in the social contract around website hosting and AI-powered scraping.
A no-JS solution is currently in development but is not yet viable. Anubis requires modern JavaScript features, which can be problematic if plugins like JShelter are enabled. Users must weigh their security needs against the convenience of having JavaScript disabled for this specific domain.
As AI-powered scrapers continue to pose a significant threat to online resources, solutions like Anubis remain crucial in mitigating these risks. While not perfect, this compromise measure serves as an interim step towards more effective strategies that prioritize website security and user experience.