Screenshot a list of URLs in bulk, and watch the queue drain

Where does the list of URLs come from?

A list of URLs for bulk screenshots comes from one of three places: a spreadsheet you already have, the site's sitemap, or a crawl of the site. All three end as the same file: urls.txt, one address per line.

1. 1A spreadsheet or CSV column

   You already have the addresses: a client list, a directory, an export from analytics or a CMS.

   • Export the sheet as CSV, keep the address column, and write one URL per line to urls.txt.
   • The one-liner drops the header row, quotes and Windows line endings, and removes exact duplicates. It assumes no comma inside the address; if yours have one, export that single column on its own.

   # pages.csv has a header row and the address in column 1
   tail -n +2 pages.csv | cut -d, -f1 | tr -d '"\r' | sort -u > urls.txt

2. 2A sitemap.xml

   You want every page a site lists for search engines, which is usually every page that matters.

   • A sitemap is an XML file with a <urlset> root, one <url> per page and the address in <loc>. One file holds at most 50,000 URLs and 50 MB uncompressed, and may be gzipped.^[1]
   • Larger sites publish a sitemap index: a <sitemapindex> root whose <sitemap> entries each point at another sitemap in <loc>, up to 50,000 of them.
   • The address of the sitemap is often in robots.txt, on a line that starts with Sitemap:. Values in the file are entity-escaped, so & has to become & again.

   // node sitemap.mjs https://example.com/sitemap.xml > urls.txt
   import { gunzipSync } from 'node:zlib'
   
   const decode = (text) =>
     text.replace(/&lt;/g, '<').replace(/&gt;/g, '>').replace(/&quot;/g, '"').replace(/&apos;/g, "'").replace(/&amp;/g, '&')
   
   async function locs(sitemapUrl) {
     const response = await fetch(sitemapUrl)
     if (!response.ok) throw new Error(`${sitemapUrl}: HTTP ${response.status}`)
     let bytes = Buffer.from(await response.arrayBuffer())
     if (bytes[0] === 0x1f && bytes[1] === 0x8b) bytes = gunzipSync(bytes) // a .xml.gz file
     const xml = bytes.toString('utf8')
     const found = [...xml.matchAll(/<loc>\s*(.*?)\s*<\/loc>/gs)].map((match) => decode(match[1]))
     if (!/<sitemapindex[\s>]/.test(xml)) return found
     const all = [] // a sitemap index lists other sitemaps: read each one in turn
     for (const child of found) all.push(...(await locs(child)))
     return all
   }
   
   console.log([...new Set(await locs(process.argv[2]))].join('\n'))

3. 3A crawl of your own site

   There is no sitemap, or you want the pages people can actually reach by clicking.

   • A crawl starts at one page, collects the links that stay on the same origin, and repeats for each new page until it has seen them all or hits a cap.
   • Read /robots.txt first and leave out what it disallows. This sketch does not parse it, which is acceptable only on a site you run. It pauses a second between requests and stops at 500 pages.^[2]
   • A crawl finds only pages that are linked. A sitemap finds only pages that are listed. For a full inventory, merge the two lists and deduplicate.

   // node crawl.mjs https://example.com > urls.txt   (a site you run)
   const start = new URL(process.argv[2])
   const MAX_PAGES = 500
   const PAUSE_MS = 1000 // one request a second to the one host
   
   const seen = new Set([start.href])
   const queue = [start.href]
   const pages = []
   for (let i = 0; i < queue.length && pages.length < MAX_PAGES; i++) {
     const response = await fetch(queue[i], { headers: { 'User-Agent': 'site-inventory ([email protected])' } })
     if (response.ok && (response.headers.get('content-type') || '').includes('text/html')) {
       pages.push(queue[i])
       for (const [, href] of (await response.text()).matchAll(/<a\s[^>]*?href=["']([^"'#]+)/gi)) {
         let link
         try { link = new URL(href, response.url) } catch { continue }
         if (link.origin === start.origin && !seen.has(link.href)) { seen.add(link.href); queue.push(link.href) }
       }
     }
     await new Promise((resolve) => setTimeout(resolve, PAUSE_MS))
   }
   console.log(pages.join('\n'))

A bulk screenshot script in Node.js, with no packages

This Node.js script screenshots every URL in urls.txt with a fixed number of requests at once, a per-minute limit, retries with backoff and a results file. It uses only the standard library and needs Node.js 18 or later, where fetch is built in.^[6]

// node bulk.mjs   (Node.js 18 or later, no packages)
import { mkdir, readFile, writeFile } from 'node:fs/promises'
import { createHash } from 'node:crypto'

const API = 'https://curlshot.com/api/v1/screenshot'
const KEY = process.env.SCREENSHOT_ACCESS_KEY
const CONCURRENCY = 4 // renders at once
const PER_MINUTE = 60 // requests per minute
const TRIES = 4

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
const lines = (await readFile('urls.txt', 'utf8')).split(/\r?\n/).map((line) => line.trim())
const urls = [...new Set(lines.filter(Boolean))]

// Readable and unique: the host and path, plus 8 characters of a hash of the whole URL.
function fileName(url) {
  const { hostname, pathname } = new URL(url)
  const slug = (hostname + pathname).toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '')
  return `${slug.slice(0, 80)}-${createHash('sha256').update(url).digest('hex').slice(0, 8)}.png`
}

// Rate limit: every request, retries included, takes the next free start time.
let nextStart = 0
async function takeTurn() {
  const at = Math.max(Date.now(), nextStart)
  nextStart = at + 60_000 / PER_MINUTE
  await sleep(at - Date.now())
}

// Retry-After is a number of seconds or an HTTP date. Without it: 1 s, 2 s, 4 s.
function retryDelay(response, attempt) {
  const header = response?.headers.get('retry-after')
  const ms = header ? (/^\d+$/.test(header) ? header * 1000 : Date.parse(header) - Date.now()) : NaN
  return ms > 0 ? ms : 1000 * 2 ** (attempt - 1) + Math.random() * 250
}

async function capture(url) {
  for (let attempt = 1; ; attempt++) {
    await takeTurn()
    const query = new URLSearchParams({ url, format: 'png' })
    const response = await fetch(`${API}?${query}`, { headers: { 'X-Access-Key': KEY } }).catch(() => null)
    if (response?.ok) {
      const file = fileName(url)
      await writeFile(`shots/${file}`, Buffer.from(await response.arrayBuffer()))
      return { url, status: 'ok', file, error: '' }
    }
    const retryable = !response || response.status === 429 || response.status >= 500
    if (!retryable || attempt === TRIES) {
      const body = response ? await response.json().catch(() => ({})) : {}
      return { url, status: 'failed', file: '', error: body.error_code || (response ? `HTTP ${response.status}` : 'network error') }
    }
    await sleep(retryDelay(response, attempt))
  }
}

await mkdir('shots', { recursive: true })
const results = []
let next = 0
async function worker() {
  while (next < urls.length) results.push(await capture(urls[next++]))
}
await Promise.all(Array.from({ length: CONCURRENCY }, worker))

const cell = (value) => `"${String(value).replace(/"/g, '""')}"`
const rows = results.map((r) => [r.url, r.status, r.file, r.error].map(cell).join(','))
await writeFile('results.csv', ['url,status,file,error', ...rows].join('\n') + '\n')
console.log(`${results.filter((r) => r.status === 'ok').length} of ${urls.length} saved. See results.csv.`)

What each part of the script does

The queue

CONCURRENCY workers share one counter, next. Each takes the next URL when it finishes the last, so exactly that many renders are open at once.

The rate limit

takeTurn hands out start times 60 / PER_MINUTE seconds apart. Retries take a turn too, so the script never sends faster than the limit.

Retries

A network error, a 429 and any 5xx are tried again, up to TRIES times. A 4xx such as a bad address or an empty quota is recorded at once: asking again cannot fix it.

Retry-After

The header is either a whole number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Fri, 31 Dec 1999 23:59:59 GMT). The script reads both, and falls back to 1, 2, 4 seconds plus a little randomness.^[3][4]

File names

The host and path become a slug of a-z, 0-9 and hyphens, cut at 80 characters, followed by 8 hex characters of the SHA-256 of the whole URL, so two addresses never share a file.

The manifest

results.csv has one row per URL with url, status, file and error. Every cell is quoted, so an address with a comma stays in one column.

The script calls the CurlShot screenshot endpoint and reads its key from the SCREENSHOT_ACCESS_KEY environment variable. For another service, change the API line and the header: the queue, the rate limit, the retries and the manifest stay the same.

How to name and store bulk screenshot files

A bulk screenshot file needs a name that is safe on every file system, unique per URL and possible to compute again from the URL alone. A slug plus a short hash gives all three.

"url","status","file","error"
"https://example.com/","ok","example-com-0f115db0.png",""
"https://example.com/pricing","ok","example-com-pricing-23a538fd.png",""
"https://example.com/old-page","failed","","navigation_failed"

The CurlShot bulk endpoint, exactly

The CurlShot bulk endpoint starts up to 100 screenshots with one POST and renders them in the background, so the queue, the retries of temporary errors and the waiting are done on our side. This is how it behaves.

curl -X POST "https://curlshot.com/api/v1/bulk" \
  -H "X-Access-Key: YOUR_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "webhook_url": "https://example.com/hooks/screenshot",
    "requests": [
      {
        "url": "https://example.com/",
        "format": "webp",
        "viewport_width": 1280,
        "viewport_height": 800,
        "block_cookie_banners": true,
        "cache": true
      },
      {
        "url": "https://example.com/pricing",
        "full_page": true,
        "format": "jpeg",
        "timeout": 60
      },
      {
        "url": "https://example.com/blog",
        "viewport_device": "iphone_15_pro",
        "wait_until": "networkidle"
      }
    ]
  }'

Start with the bulk screenshots page, read every field in the bulk endpoint reference, and see async jobs and webhooks for the signature check on each delivery. The numbers of each plan are in usage and limits.