Company · crawler
UnfragileBot
Our crawler. It reads the public websites of AI infrastructure and software companies so our search can say what each product is and does. It reads little, obeys robots.txt and signs every request.
What it reads
- A handful of public pages from each company's own site: at most seven, starting from the home page.
- One page for a named product, when we know its address.
- Once a week. Pages that only appear after JavaScript runs are read with a headless browser, Lightpanda.
- Nothing behind a login, a paywall or a form, and no tricks to get past a block or a challenge page. When a site challenges us, we may read the file it publishes for AI tools (
llms.txt) instead, and only if robots.txt allows it.
We use what we read to describe and rank products in our search.
How to recognise it
Every request names us in its user agent:
UnfragileBot/1.0 (+https://unfragile.ai/bot; AI infrastructure search index)A name can be copied, so every request is also signed with Web Bot Auth (HTTP Message Signatures, Ed25519). A signed request carries three headers: Signature-Agent (pointing to https://unfragile.ai), Signature-Input and Signature. Our public key is published at:
https://unfragile.ai/.well-known/http-message-signatures-directoryA request that claims to be UnfragileBot but carries no valid signature from that key is not from us. Pages read by the headless browser are not signed yet; they carry the same user agent.
How to opt out
Add this to your site's robots.txt. The next weekly run stops reading your site:
User-agent: UnfragileBot
Disallow: /You can also disallow only some paths. We read robots.txt before every site, every week.
Contact
Questions about the crawler, or something we read that we should not have: arham@unfragile.ai