Ask HN: How do you protect self-hosted websites from LLM scrapers?

Hello HN,

I'm considering self-hosting a couple of static low-volume traffic on a VM. I'm curious to know what techniques you using to protect personal and professional websites from abusive LLM (and other) scraping. Assume no CDN is involved.

I'm aware of:

    (1) anubis AI firewall
    (2) LLM poisoning projects like iocaine
    (3) bate links hidden in the HTML source code that block IPs when accessed
    (4) ppl blocking CIDRs of all major Cloud Providers (GCP, AWS, Azure, Alibaba, etc).
Are there any other interesting techniques that can be used?

Ty!

1 points | by atmosx 4 hours ago

2 comments

  • arbol 38 minutes ago
    Why do you expect a high level of abuse? Blocking CIDRs of major cloud providers will stop the genuine ChatGPT/Claude traffic. For the other scrapers masquerading as them, just block the agents' user agents.

    If you want the LLMs to refuse to scrape your site you could inject lots of encoded trigger words but this would probably affect your SEO score :)

  • selfhoster11 4 hours ago
    Why would you not put a CDN in front? I’d rather do that than CAPTCHA.
    [-]
    • atmosx 2 hours ago
      Because it’s a third party service, plus CDNs have Captcha anyway.