SitesRadarBot

User-agent strings

We identify ourselves with two user-agent strings, depending on the task:

Page crawling & health checks

SitesRadarBot/1.0 (+https://sitesradar.com/bot)

Alert delivery pings (when we verify a site is back up before sending you a recovery notification)

SitesRadarAlerts/1.0 (+https://sitesradar.com)

What we fetch

When a site is checked — either through the free checker or as part of a paid monitoring subscription — our crawler:

• sends a standard HTTP GET request to the site's URL;
• follows same-domain links to discover pages (BFS, up to 10 pages for free checks, up to 50 for paid monitoring);
• reads HTML to find broken links, redirect chains, and SEO metadata;
• fetches linked documents (PDFs, images, Office files) to inspect their metadata for accidental leaks;
• checks the SSL certificate chain and domain registration status via public registries (RDAP).

We do not execute JavaScript. All requests are server-side HTTP fetches — what a browser would receive before any client-side rendering runs.

How we behave

• Politeness: at least 300 ms between consecutive requests to the same host.
• Concurrency: no more than 3 simultaneous requests to a single host.
• Page limit: 10 pages per free check, 50 per paid monitoring sweep.
• Scope: same-domain only — we do not crawl external sites linked from your pages.
• Free checker rate limit: 5 checks per hour per IP address.
• Timeouts: each request times out after 8 seconds.

Security

Our fetcher is SSRF-guarded: it will not follow links to private IP ranges (10.x, 192.168.x, 127.x), link-local addresses (169.254.x, including cloud metadata endpoints), or other internal addresses. This prevents the crawler from being used to probe internal networks.

How to block us

If you'd prefer that SitesRadarBot not crawl your site, you have two options:

1. Block by user-agent in your web server or firewall config:

# nginx
if ($http_user_agent ~* "SitesRadarBot") {
    return 403;
}

# Apache (.htaccess)
SetEnvIfNoCase User-Agent "SitesRadarBot" bad_bot
Deny from env=bad_bot

2. Contact us and we'll exclude your domain from our checker and any monitoring subscriptions that reference it.

Email info@sitesradar.com with the domain and we'll take care of it within one business day.

Questions?

If you're a site owner and have questions about a request you saw from SitesRadarBot in your logs, write to info@sitesradar.com and we'll explain what triggered the check.