ClaimFinderBot
ClaimFinderBot collects publicly posted class action settlement information so that people can find settlements they are entitled to claim. It reads claims administrator case lists and official settlement websites, honors robots.txt on every request, and makes at most one request every three seconds to any single domain. If it is causing you a problem, the fastest fix is below, and we would rather you told us than blocked us silently.
How to identify it
| User agent | ClaimFinderBot/0.1 (class action settlement directory; + https://claim-finder.com/bot) |
|---|---|
| robots.txt token | ClaimFinderBot |
| Request rate | One request per three seconds per domain, at most |
| Frequency | Once daily, early morning United States Eastern time |
| Operated by | Claim Finder |
What it does
- Reads the case list an administrator publishes, then fetches each settlement website linked from it.
- Requests only normal pages. It does not submit forms, does not attempt to log in, does not follow claim filing flows, and never enters data anywhere.
- Extracts the claim deadline, payout, eligibility, and whether proof is required, then links readers to your official site to file.
- Re-checks pages it already knows about so that stale deadlines are corrected rather than left standing.
How it behaves
- robots.txt is honored on every request, following RFC 9309: the most specific user agent group wins, the longest matching rule wins, Allow beats Disallow on a tie, and Crawl-delay is respected when it is longer than our own interval.
- One request per three seconds per domain, scheduled per host, so concurrent work never bursts against one site.
- Timeouts and backoff. Requests time out after 20 seconds, and transient failures back off exponentially. A 429 or a Retry-After header is obeyed.
- Dead ends are remembered. A domain that stops resolving or returns 403 is recorded and not re-attempted for days or weeks, so a site that does not want us does not keep getting asked.
How to block it
Add this to your robots.txt and it will stop on the next run:
User-agent: ClaimFinderBot
Disallow: /To slow it down instead of blocking it, set Crawl-delay. To keep it out of one section, disallow that path. All of it is honored.
Removing a settlement
If a page here is wrong, out of date, or you administer a settlement and want the listing corrected or removed, contact us and it will be handled. We link to official settlement websites and send readers to them to file; we do not host claim forms or collect anything from claimants.
Contact details are being set up. Until then, blocking via robots.txt takes effect on the next daily run.
See also how we verify settlements and the data behind the directory.