Merging Branch 'ps/ci-test-aggreg-fix-for-meson': The Unseen Struggle Against AI Scraping
As you read this, it's likely that the administrator of our website has taken steps to protect its servers from the scourge of AI companies aggressively scraping websites. This measure, known as Anubis, is a compromise between security and accessibility.
Anubis uses a Proof-of-Work scheme, similar to Hashcash, a proposed solution for reducing email spam. The idea behind it is that at individual scales, the added load is insignificant, but when used by multiple scraper sites simultaneously, it becomes prohibitively expensive for them to continue their activities. In essence, Anubis serves as a temporary placeholder measure, allowing developers more time to work on fingerprinting and identifying headless browsers (such as those using font rendering techniques) – making it possible to bypass the challenge without presenting users with an unsightly proof-of-work page.
However, Anubis comes with its own set of requirements. It necessitates the use of modern JavaScript features, which plugins like JShelter will disable. If you're using such plugins on our domain, we kindly request that you disable them to overcome this hurdle. Moreover, unfortunately, Anubis demands that users enable their browsers' JavaScript functionality in order to bypass it.
It's worth noting that the implementation of a no-JS solution is still a work-in-progress. The social contract around how website hosting works has shifted due to the rise of AI companies, and these measures are an attempt to adapt to this new landscape.
We appreciate your patience and understanding in dealing with these temporary challenges. Our team is working diligently to ensure that our website remains accessible while also protecting itself from malicious activities.