The renderer as a bot

Who runs the screenshot renderer, why it visits a site, how it signs its requests (Web Bot Auth), where its key directory is, and how a site owner can block it.

View as Markdown

When someone asks CurlShot for a screenshot of your site, a browser on our servers opens the page. This page says who that is, so that you can recognise the visit and decide what to do about it.

Every request the renderer sends carries these three headers:

HTTP

Signature-Agent: "https://curlshot.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1735689600;keyid="<key thumbprint>";alg="ed25519";expires=1735689660;nonce="...";tag="web-bot-auth"
Signature: sig1=:...:

#Who it is

Operator: the company named in the terms. It runs the CurlShot screenshot API (https://curlshot.com).
What it is: a headless Chromium browser, started per request. It is not a crawler: it never follows links on its own and never builds an index. It loads the one address an API user sent, plus what that page needs to draw itself (scripts, styles, images, fonts), takes a picture or a PDF, and closes.
Who starts a visit: always a person or program using the CurlShot API with their own account. CurlShot does not pick pages to visit. A visit is the same as that person opening your page in a browser.
Rate: one page load per screenshot. Every account has rate and concurrency limits, so one account cannot hit a site hard.
Contact: [email protected] or the contact page. Tell us the address of your site and what you saw.

#How to recognise it

The renderer sends the User-Agent of a current desktop Chrome on Linux (Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version> Safari/537.36). It matches every Chrome, so do not rely on it. The reliable mark is the signature below.

The headers follow Web Bot Auth (HTTP Message Signatures, RFC 9421, with the IETF drafts draft-meunier-web-bot-auth-architecture and draft-meunier-http-message-signatures-directory):

The signature covers the host the request goes to (@authority) and the Signature-Agent header, and is valid for 60 seconds.
Signature-Agent names the origin that publishes the public key: https://curlshot.com.
The public key is in a signed key directory at https://curlshot.com/.well-known/http-message-signatures-directory (a JSON Web Key Set, Ed25519, served as application/http-message-signatures-directory+json). A request is ours if its signature verifies with a key in that directory.

There is no published list of addresses: the renderer runs on machines whose addresses can change. Use the signature.

Anyone can imitate the User-Agent or the headers; nobody can forge the signature without the private key. Cloudflare and other services that support Web Bot Auth can verify these signatures (see verified bots).

#What it does and does not do

It identifies itself on every request, whatever the host and resource type.
It does not solve CAPTCHAs, click through "verify you are human" pages or work around a challenge. A page that answers with a bot check is not captured: the API user gets the page_blocked error and is not charged.
It does not read robots.txt: like a link preview or a person's browser, it fetches one address a user asked for. If you do not want it on your site, block it as below.
It only visits public addresses. Private networks, localhost and cloud metadata addresses are refused before any connection.

#How to block or allow it

Block by signature. Match the Signature-Agent request header (its value is "https://curlshot.com"). In a firewall or WAF rule, block requests where that header is present, or verify the signature against the key directory above and block on the keyid.
Allow it on your own site. If you are the owner and your bot protection blocks your own screenshots, use one of the rules under Allow it on your own site.
Ask us. Write to [email protected] with your domain and we will look at it. See also the acceptable use policy.

#Allow it on your own site

Your own bot protection can stop screenshots of your own pages: the API answers page_blocked. No setting on our side changes that, and the renderer does not solve the check. You let it in, with one rule placed before the rule that blocks or challenges. Three ways to match it, strongest first:

A verified signature. Where your provider verifies Web Bot Auth for you, allow the verified bot. Nothing can imitate it.
A secret header of your own. Send a header only you know with the headers option, for example X-Capture-Key: <a long random value>, and allow requests that carry it. It works on every firewall and does not depend on us. Treat the value like a password.
The Signature-Agent header. Its value is always "https://curlshot.com", quotes included. Simple, but a header on its own can be copied by anyone, so use it only for pages that are public anyway.

Security > WAF > Custom rules > Create rule

Expression (verified by Cloudflare, and ours):
  (cf.client.bot and any(http.request.headers["signature-agent"][*] eq "\"https://curlshot.com\""))

Action: Skip
  Skip: all remaining custom rules, rate limiting rules,
        managed rules, Super Bot Fight Mode rules

Place it first in the list.

Web ACL > Rules > Add my own rules > Rule builder

If a request matches the statement
  Inspect:     Single header
  Header name: signature-agent
  Match type:  Exactly matches string
  String:      "https://curlshot.com"

Action: Allow
Priority: before the Bot Control rule group

Project > Firewall > Configure > New rule

If
  Request Header   signature-agent   Equals   "https://curlshot.com"
Then
  Bypass

Place it above the rules that challenge or deny.

# 1 when the request names our key directory, else 0
map $http_signature_agent $capture_bot {
    default            0;
    '"https://curlshot.com"'   1;
}

# then exempt it where you limit or challenge, for example:
map $capture_bot $limit_key {
    0 $binary_remote_addr;
    1 "";                      # an empty key is not rate limited
}
limit_req_zone $limit_key zone=perip:10m rate=5r/s;

A CAPTCHA widget that is part of the page itself (reCAPTCHA, hCaptcha or Turnstile on a form) is drawn like any other element: it appears in the screenshot and is not solved. If a full-page check from one of those services stands in front of your site, its dashboard is where you exempt a visitor, with the same three marks.