Merge branch 'rj/build-tweaks': The Battle Against Bot Scraping

The administrator of this website has taken steps to protect its server from the relentless onslaught of AI-powered web scrapers, a scourge that threatens to drain resources and render websites inaccessible to everyone. To combat this menace, Anubis has been deployed as a security measure, a compromise solution that requires users to complete a simple yet clever challenge.

Anubis employs a Proof-of-Work scheme, inspired by Hashcash, a proposed method for reducing email spam. At the individual scale, the added load is negligible, but when scaled up to mass scraper levels, it becomes a significant hurdle. The idea behind Anubis is to make scraping more expensive and less appealing, thereby discouraging AI companies from targeting websites with malicious intent.

However, the true intention of Anubis goes beyond mere protection against bot scrapers. Its primary goal is to provide a "good enough" placeholder solution that allows developers to focus on fingerprinting and identifying headless browsers – those stealthy, AI-powered browser emulators that can mimic human behavior. By presenting users with the challenge proof-of-work page, Anubis aims to weed out legitimate visitors who are more likely to use JavaScript, thereby ensuring a smoother experience for genuine users.

But Anubis comes with some caveats. It requires the use of modern JavaScript features, which plugins like JShelter may disable. To bypass this challenge, users must enable JavaScript, a necessary evil in today's web landscape where AI companies have rewritten the social contract around website hosting. While a no-JS solution is still in development, Anubis serves as a temporary measure to safeguard websites from the threat of bot scraping.

As we navigate this complex digital landscape, it's essential to acknowledge the trade-offs involved in deploying security measures like Anubis. While they may inconvenience users, they ultimately help preserve the integrity and functionality of our online experiences. So, the next time you encounter a challenge like Anubis, remember that it's all part of the ongoing battle against bot scrapers and the quest for a safer, more secure web.