Merge branch 'es/meson-configure-build-options-fix': Protecting Against Aggressive Scraping
As you read this article, it's likely because the administrator of this website has taken steps to protect against a very real threat: aggressive scraping by AI companies. This measure is implemented using Anubis, a system designed to prevent these unwanted visitors from accessing our resources.
Anubis operates on a Proof-of-Work scheme, similar to Hashcash, which aims to discourage mass scrapers by making the task significantly more expensive. At an individual level, this may seem negligible, but when applied at scale, it deters malicious activity effectively.
The primary goal of Anubis is not to eliminate scraping entirely, but rather to provide a "good enough" placeholder solution that buys time for more advanced solutions. Its real purpose lies in improving the ability to fingerprint and identify headless browsers (like those used by AI companies) without presenting users with an unnecessary challenge.
However, Anubis comes with a caveat: it requires modern JavaScript features, which may be disabled by certain plugins like JShelter. If you're accessing this site, it's essential to disable such plugins or ensure they are configured to work around the Anubis system.
"This is a necessary evil," we must acknowledge. The threat posed by AI companies has rewritten the social contract surrounding website hosting. By requiring JavaScript to bypass Anubis, we're forced to accept that no longer will websites be protected solely by static content or outdated security measures.
While a fully JavaScript-based solution remains a work-in-progress, Anubis stands as a crucial step in the ongoing fight against aggressive scraping and AI-driven abuse. By understanding its purpose and limitations, we can better navigate these evolving threats and ensure the continued integrity of our online world.