How to Run a Technical SEO Audit With Screaming Frog Without Missing What Matters

Screaming Frog's SEO Spider is a genuinely thorough technical crawler, and it's also a tool with default settings that quietly fail on exactly the sites that need a technical audit most: large sites that overwhelm default memory allocation, and JavaScript-heavy sites that render completely differently to a crawler than they do to a browser unless rendering is explicitly turned on. A crawl run on defaults against either of those sites produces a report that looks thorough and misses the majority of what's actually wrong. Here's the setup that avoids that.

For background on what Screaming Frog is, its free and paid tiers, and how it fits alongside a broader SEO platform, see our full Screaming Frog profile.

Step 1: Configure Memory Allocation to Match the Site's Actual Size

Screaming Frog's default memory allocation is tuned for a small crawl and starts struggling well before it reaches even a moderate site size, commonly around the tens of thousands of URLs. Before crawling anything beyond a small brochure site, increase the memory allocation in the tool's configuration to match the site's rough scale, and for very large sites, switch to database storage mode rather than the default memory-based storage. Skipping this step on a large site doesn't just risk a slow crawl, it risks the crawl freezing or crashing partway through, silently truncating the audit to whatever portion completed before it died, with no obvious signal that the resulting report is incomplete.

Step 2: Decide on JavaScript Rendering Deliberately, Not by Default

If the site is built on a JavaScript framework like React, Angular, or Vue, where meaningful content or navigation renders client-side, set the crawl's rendering mode to JavaScript rather than the default text-only crawl. Without it, the crawler sees only the initial server-rendered HTML and misses whatever content, links, or metadata gets injected by client-side JavaScript, which produces a crawl that looks complete but is blind to a large share of the actual page. This comes at a real cost: JavaScript rendering uses a full browser engine under the hood and slows the crawl substantially, so it's worth confirming the site actually needs it (checking whether meaningful content differs between viewing page source and the rendered DOM) rather than defaulting to it for every crawl regardless of the site's architecture.

Step 3: Triage by Response Code Before Looking at Anything Else

Once the crawl completes, start with response codes: server errors, broken links, and redirect chains, before moving into more granular on-page analysis. These are the highest-severity issues a crawl surfaces, actual broken user experiences and wasted crawl budget, and fixing them first is both the highest-leverage use of audit time and the check that should happen before investing effort in lower-severity issues like meta description length on pages that might not even be reachable without a broken redirect chain in the way.

Step 4: Check Crawl Depth and Orphan Pages, Not Just Broken Links

Beyond response codes, review the crawl's page depth data (how many clicks from the homepage a given page sits at) and cross-reference the crawled URL list against a sitemap or CMS export to identify orphan pages, pages that exist but have no internal links pointing to them at all. Pages buried many clicks deep, or entirely orphaned from the site's internal link structure, are frequently under-indexed and under-ranking not because of any content problem but because internal linking never gave them a fair chance to be discovered and crawled efficiently. This is one of the more commonly overlooked findings in a technical audit precisely because it doesn't show up as an error, a page with zero internal links crawls and renders fine, it's just invisible to the site's own linking structure.

Step 5: Audit Duplicate and Missing Metadata at Scale

Use the built-in duplicate content and duplicate title/meta description reports to find pages competing against each other or missing basic on-page signals entirely, rather than spot-checking a handful of pages manually. At scale, this kind of issue is common on templated content, category pages, or paginated series that inherited the same title pattern without anyone noticing until a crawl aggregates the pattern across hundreds of URLs at once.

Step 6: Connect Search Console and GA4 to See Crawl Data Alongside Real Performance

Screaming Frog integrates directly with Google Search Console, GA4, and PageSpeed Insights, pulling that data into the same crawl report rather than requiring a separate cross-reference. This combined view catches a category of issue a crawl alone can't: a page that crawls perfectly clean, no errors, solid metadata, but has near-zero organic impressions or clicks according to Search Console, which points toward a content, relevance, or indexing issue a purely technical crawl wouldn't flag on its own. Conversely, a page driving meaningful real traffic that also shows a technical issue in the crawl is a much higher priority fix than the same issue on a page nobody visits.

Step 7: Use Custom Extraction for Anything the Default Reports Don't Cover

For site-specific checks the built-in reports don't handle, validating a specific schema markup implementation, confirming a CMS-injected tag is present sitewide, pulling a specific custom field, use custom extraction with CSS or XPath selectors to pull that exact data point across the whole crawl. This is what turns Screaming Frog from a generic auditing tool into one scoped to the specific technical questions a given site actually needs answered, rather than limited to whatever issues the default report categories happen to cover.

Step 8: Prioritize the Export by Combining Severity and Page Value

Before handing off the audit's findings, weight the exported issue list by both technical severity and the actual value of the affected page, using the traffic and performance data pulled in Step 6, rather than treating every broken link or duplicate title as equally urgent. A broken link on a high-traffic landing page and the same broken link on a page nobody visits are not the same priority, even though a raw crawl export lists them identically, and a fix list ordered by actual business impact gets acted on faster than one ordered purely by issue type.

A Few Mistakes That Undo This Workflow

A handful of habits reliably produce an incomplete or misleading audit even with a technically successful crawl. Running every crawl with JavaScript rendering enabled by default, regardless of whether the site actually needs it, wastes significant time on sites that don't require it and can introduce its own rendering quirks that complicate interpretation. Treating a completed crawl as a finished audit without checking whether it actually reached expected completion, silently truncated by a memory limit partway through, means a large site's audit can look thorough while having actually covered only a fraction of the site. And exporting the full issue list to a client or stakeholder without the severity-and-value prioritization from Step 8 hands over a wall of data that reads as overwhelming rather than actionable, which is how a genuinely useful audit ends up shelved instead of acted on.

What This Workflow Actually Produces

Run this way, a Screaming Frog audit stops being a raw export of every technical issue a crawl happened to find and becomes a genuinely prioritized diagnosis: configured correctly for the site's actual size and architecture, triaged by real severity, cross-referenced against actual performance data, and ordered by what's actually worth fixing first. That's a meaningfully more useful deliverable than a spreadsheet of every duplicate title tag on the site with no sense of which ones matter, and it's the difference between a technical audit that changes what gets fixed first and one that gets skimmed once and filed away.

None of this requires the paid license to start, the free tier's 500-URL cap still supports every step above on a smaller site, and the same discipline, correct configuration, severity triage, cross-referencing real performance data, scales up cleanly the moment a larger site or the paid license's unlimited crawl size actually becomes necessary.