KatlaBot
The crawler Katla uses to scan websites for cookies and trackers, on behalf of the site's owner. It signs every request it sends and follows robots.txt.
What it does
KatlaBot is the crawler behind Katla's site scans. It visits a site when a Katla customer has added that site to their account, either because they started a scan or on the schedule their plan includes. It finds the cookies and tracking technologies the site uses, so the site's cookie banner and cookie policy can disclose them.
On each page it loads, it:
- renders the page in a real browser (headless Chrome), as a visitor would see it;
- accepts the site's own cookie banner, because most trackers only load after consent and a scan that never consents would miss them;
- records the cookies that were set and the third parties the page contacted;
- runs an automated accessibility check;
- follows links to other pages on the same site, and nowhere else.
It does not fill in forms, sign in, or buy anything.
How to verify it
KatlaBot signs its requests with HTTP Message Signatures (RFC 9421), as described by Web Bot Auth. Every request it sends to the site it is scanning carries three headers:
Signature-Agent: "https://katla.app"
Signature-Input: sig1=("@authority" "signature-agent");created=...;expires=...;keyid="...";alg="ed25519";nonce="...";tag="web-bot-auth"
Signature: sig1=:...:The public keys are published at https://katla.app/.well-known/http-message-signatures-directory. A request is from KatlaBot only if its signature verifies against one of them.
Cloudflare does this check for the sites behind it: KatlaBot is a verified bot in Cloudflare's bots directory.
Verify by signature, not by address or user agent. KatlaBot does not crawl from a fixed set of IP addresses, and pages are loaded with an ordinary desktop Chrome user agent, because a cookie scan has to see what a visitor's browser is served. robots.txt itself is fetched with this user agent:
Mozilla/5.0 (compatible; KatlaBot/1.0; +https://katla.app/bot)
Requests the page makes to your own site while it renders, such as scripts, styles and API calls, are signed too. Requests to third parties are not.
robots.txt
KatlaBot reads robots.txt before it loads any page, separately for each host, so example.com and www.example.com each follow their own. It obeys the group for User-agent: KatlaBot if there is one, and the User-agent: * group otherwise.
DisallowandAllowdecide which pages it loads, including pages it is redirected to.Crawl-delayis honoured: it loads one page at a time, at least that many seconds apart. A long delay means it scans fewer pages rather than taking longer.- If robots.txt answers with a server error or cannot be reached, it loads nothing. If there is no robots.txt, it may load any page.
robots.txt decides which pages KatlaBot visits. Once a page is loaded, the resources that page asks for are fetched as a browser would fetch them.
To keep KatlaBot off your site:
User-agent: KatlaBot Disallow: /
To let it scan a site whose robots.txt otherwise keeps crawlers out:
User-agent: KatlaBot Allow: /
How hard it crawls
KatlaBot loads a few pages at a time at most, and stops at the page limit of the customer's plan. On sites behind Cloudflare it waits at least three seconds between page loads from the first page on.
If a site answers with a rate limit or a bot challenge, KatlaBot backs off: it waits, continues one page at a time with a longer gap, and ends the scan if the site keeps pushing back. The customer is told the site blocked the scanner.
Letting it through bot protection
KatlaBot is a verified bot in Cloudflare's bots directory, and Cloudflare's default bot settings leave verified bots alone. Most sites behind Cloudflare have nothing to change.
If a Katla scan still reports that your site blocked the scanner, one of your own settings stopped it. In the Cloudflare dashboard, open Security, then Analytics, and look at the Events tab around the time of the scan: each event names the feature or rule that acted on the request. The usual causes are a custom rule that challenges visitors broadly, a rate limiting rule, or Super Bot Fight Mode set to block verified bots.
One custom rule lets verified bots past all three. Create it on the Security rules page, under Create rule, then Custom rules, and put it first in the list, because it only skips the rules that come after it:
Expression: (cf.client.bot)
Action: Skip
Skip: All remaining custom rules
All rate limiting rules
All Super Bot Fight Mode rulesThis lets every Cloudflare verified bot through, search engines included, not only KatlaBot. To exempt verified bots from a single rule instead, add and not cf.client.bot to that rule's expression. In the rule builder the field is called Known Bots.
If you cannot find these settings, your host may run Cloudflare for you, on its own account. Ask the host to let Cloudflare verified bots through on your site, and send them this page.
With other bot protection, look for a setting that allows verified bots or requests signed with Web Bot Auth. KatlaBot has no fixed IP addresses to allowlist.
Questions or problems
If KatlaBot is causing trouble on your site, or you want to know why it visited, contact us and include the site's address and the time of the visit.