Merge branch 'dd/meson-perl-custom-path': The Unseen Battle Against AI Scraping
As you navigate through websites today, it's likely that you've encountered an unusual obstacle: a challenge that requires you to prove you're not a bot. This may seem like a minor inconvenience, but for website administrators and developers, this "feature" – known as Anubis – poses significant challenges to their online presence.
So, what exactly is Anubis? In simple terms, it's an anti-bot measure designed to protect websites from AI companies that aggressively scrape them. The administrator of this website has set up Anubis as a way to shield themselves against these malicious activities, which can cause downtime and make resources inaccessible to everyone.
Anubis uses a Proof-of-Work scheme, similar to Hashcash, to deter mass scrapers. While it may seem like an insignificant burden at the individual scale, when applied to large-scale AI companies, this added load becomes prohibitively expensive for them to overcome.
But what's the real purpose of Anubis? It serves as a temporary hack to give website administrators more time to work on fingerprinting and identifying headless browsers. By delaying these legitimate users from accessing their websites, Anubis essentially presents its challenge proof-of-work page only to those who are most likely to be malicious – AI companies.
However, there's a catch: Anubis requires the use of modern JavaScript features, which plugins like JShelter will disable. To proceed past this challenge, you must enable JavaScript, a requirement that stems from the changing social contract around website hosting and AI scraping.
"A no-JS solution is still in development," according to the current state of affairs. In other words, there's an ongoing effort to find alternative solutions for websites without relying on JavaScript – but this work remains incomplete for now.