SiteDecayBot
Saw SiteDecayBot in your logs? It's Site Decay's research crawler. We measure how websites age: expiring certificates, outdated software, broken pages, and sites that go offline or get hacked.
What it reads
One visit per site per crawl. A visit is:
- /robots.txt, read first. If it keeps SiteDecayBot (or every bot) off your homepage, we stop there.
- Your homepage: a single GET request that follows redirects and reads at most 1.5 MB.
- Your RSS or Atom feed, only if your homepage links to one, to see the date of your newest post.
- Your certificate: one TLS handshake on port 443, the same check a browser makes.
Nothing else. No other pages, no forms, no logins, no admin paths, and no ports other than 80 and 443.
How often
A crawl usually comes about once a week, often less. Each one visits a site once, a few seconds of traffic at most.
What we do with it
Our findings are published as group counts, like "how many sites still run PHP 7" or "how many certificates expired this month". We never publish a site by name as hacked, broken or out of date.
Opting out
Either way works, and both take effect from the next crawl.
In robots.txt, add:
User-agent: SiteDecayBot
Disallow: /Or tell us here and we'll leave the site out:
Identifying it
The user agent is:
Mozilla/5.0 (compatible; SiteDecayBot/1.0; +https://sitedecay.com/bot)Questions or a problem with a visit? Contact us. A person reads every message.