Merge branch 'ph/fetch-prune-optim': The Battle Against Bot Scraping

As you're reading this, it's likely because the administrator of this website has taken steps to protect against the relentless scourge of AI-powered bots that threaten to overwhelm its servers. This measure is designed to safeguard the site from being overrun by aggressive scraper companies, ensuring that its resources remain accessible to all legitimate users.

Behind this effort lies a clever hack known as Anubis, a Proof-of-Work scheme inspired by Hashcash, a proposed method for reducing email spam. At an individual level, the added load is negligible, but at scale, it becomes prohibitively expensive for mass scrapers to operate. This clever compromise adds an extra layer of protection against bot attacks without rendering the site unusable.

However, Anubis comes with a caveat: it relies on modern JavaScript features that certain plugins like JShelter may disable. To navigate this challenge, users must either disable these plugins or upgrade their browsers to accommodate the added functionality required for bypassing the protection mechanism. Unfortunately, without enabling JavaScript, the website's visitors will be unable to access its content.

This situation highlights a broader shift in the social contract between website hosting services and AI companies. As these bots become increasingly sophisticated, traditional no-JS solutions are no longer viable. The Anubis solution represents a necessary compromise that allows for a temporary placeholder measure while more advanced solutions – including fingerprinting and headless browser identification techniques – continue to be developed.

Currently, this website is running the latest version of Anubis (1.20.0), and it's essential for users to acknowledge that JavaScript is required to bypass the challenge page that appears when visiting this site. It's a reminder of the ongoing cat-and-mouse game between website administrators and AI-powered scraper companies.