Lyre/technical/bans/fail2ban.md

Fail2Ban - Automated Banning for Persistent Scrapers

Fail2Ban is a log-monitoring and IP-banning framework that automatically detects and blocks repeat offenders based on configurable patterns in server logs. In the context of AI scraper defense, it serves as the enforcement layer that turns repeated violations of robots.txt and tarpit hits into temporary or long-term IP bans.

Why Fail2Ban Matters

Even with strong first-line defenses like Anubis (proof-of-work) and Nepenthes (tarpits), a small number of sophisticated or persistent scrapers may continue probing or rotating through infrastructure. These actors impose recurring costs. Fail2Ban shifts the burden: after a configurable number of violations (e.g., five hits on a tarpit path within a short window), the offending IP is automatically banned for 24 hours or more.

This creates a strong deterrent signal. Scrapers quickly learn that repeated policy violations result in self-inflicted outages, making the economic cost of continued scraping higher than the value of the stolen data.

How It Fits the Defense Stack

1. Anubis (anubis.md) - First filter (computational challenge).
2. Nepenthes (nepenthes-tarpit.md) - Second filter (resource-wasting tarpit).
3. Active denial techniques - Third layer (bombs, malformed content, slowloris).
4. Fail2Ban (this document) - Enforcement layer that automatically escalates repeated violations into IP bans.
5. UA reference list (known-aggressive-bot-user-agents.md) - Shared intelligence used to build the ban rules.

Fail2Ban is the "teeth" of the system. It ensures that the passive layers are not merely advisory but carry real, automated consequences.

Key Benefits for Individuals

• Zero manual intervention - Once configured, bans are applied and lifted automatically.
• Highly configurable - Ban duration, retry thresholds, and log patterns can be tuned per path (tarpit, malformed, protected).
• Log-driven intelligence - Uses the same ai_violators.log already generated by the aggressive-bot map.
• Lightweight - Runs as a simple daemon with negligible resource impact.
• Open source - Mature, widely trusted, and actively maintained.

Recommended Integration

Configure Fail2Ban jails to watch the dedicated violator logs produced by Anubis, Nepenthes, and the conditional serving logic. A typical rule might ban any IP that hits /tarpit/ or /malformed/ more than five times in ten minutes. This keeps the system responsive while avoiding false positives on legitimate research traffic (which is already whitelisted via reverse-DNS in the UA list).

Official Resources

• Project: https://github.com/fail2ban/fail2ban
• Documentation: https://fail2ban.readthedocs.io/

Recommended Starting Point

After Anubis and Nepenthes are in place, add Fail2Ban as the final automated enforcement mechanism. It transforms the defense stack from a set of speed bumps into a self-policing system that protects individual creators at scale with almost no ongoing effort.

Fail2Ban completes the passive defense layer by providing automatic, log-driven enforcement. It works in concert with every other technique documented in this repository.