Merge branch 'kj/glob-path-with-special-char' - Protecting Against Bot Scrapers
As a precautionary measure, this website has been set up with Anubis, a security system designed to prevent AI companies from scraping websites without permission. This can result in downtime for the website, making its resources inaccessible to everyone.
Anubis is a compromise between security and usability. It uses a Proof-of-Work scheme similar to Hashcash, which aims to deter mass scrapers by adding an additional load to their operations. On an individual scale, this load may be negligible, but it becomes significant when applied to multiple scrapers, making the process much more expensive and time-consuming.
The true purpose of Anubis is not to eliminate scraping entirely, but rather to serve as a temporary solution while researchers work on developing a more sophisticated system to identify and fingerprint headless browsers. These are the types of browsers used by AI companies to scrape websites without being detected.
However, Anubis does require the use of modern JavaScript features, which can be disabled by plugins such as JShelter. If you're using one of these plugins, please disable it for this domain. Unfortunately, enabling JavaScript is still necessary to overcome the challenge posed by Anubis.
The current solution relies on JavaScript, and a no-JS solution is still in development. The social contract around website hosting has changed significantly due to the rise of AI companies, and a more secure approach is needed. Researchers are working tirelessly to develop a better system that balances security with usability.