Tools & Data
What Is Screaming Frog?
A desktop crawler that fetches a site the way a search engine would and reports what it found, one URL per row.
Definition
Screaming Frog SEO Spider is a desktop application that crawls websites and reports technical data for each URL, including status codes, titles, meta descriptions, headings, canonical tags, redirects and structured data. It runs locally rather than as a hosted service.
What it is good at
- Finding broken links, redirect chains and loops
- Auditing titles, meta descriptions and heading structure across a whole site
- Rendering JavaScript so client rendered content can be compared with the raw response
- Extracting arbitrary elements with CSS selectors or XPath
- Joining crawl data with Search Console and analytics APIs
Where the desktop model helps and hurts
Running locally means no per URL fee and full control over the crawl, which suits one off audits. It also means the crawl is bounded by the machine it runs on, and there is no continuous monitoring between runs. Large sites need database storage mode and patience, or a hosted crawler instead.
A note on comparison
A crawler shows what is reachable from your seed URL by following links. It does not show what Google actually crawled. Those two sets differ, often substantially, which is why crawl data and log data answer different questions.
A crawl tells you what a crawler could reach. Server logs tell you what a crawler chose to spend requests on. Auditing only the first leaves the more expensive problems invisible.