crawl
A crawl is a systematic pass through every page of a site to record what is actually there, used to base a proposal or an audit on real facts instead of guesses.
A crawl is a systematic pass through every page of a website, done by a piece of software that follows every link it finds and records what it sees. It is the same basic technique Google’s own crawler uses to discover and index pages, just run for the purpose of understanding a site rather than ranking it.
I run a crawl before writing almost any proposal. It tells me exactly what pages exist, what is on them, which ones are missing basic things like a title or a clear heading, and how the whole structure holds together as a network of links. Nothing in that process depends on the client granting access to any account, because a crawl works from the outside, the same way a visitor or a search engine sees the site.
The value of a crawl is that it replaces guessing with a record. When a proposal says a site has a specific number of pages, or is missing a specific technical element, that claim comes from an actual pass through the site, not an impression formed by skimming the homepage.
Why it matters to you
A proposal built on a crawl is a proposal built on your actual site, not a generic template with your business name pasted in. That matters because the fixes that make sense for one site are often wrong for another. A crawl is what lets a recommendation be specific: this page is missing this thing, this many pages have this problem, here is what fixing it looks like.
It also gives you something to check. Because a crawl is a factual record, you can ask where a claim came from, and the answer is always traceable back to a real page on your real site.
How I use it
Before I write a proposal, I crawl the existing site, whatever platform it runs on, WordPress, a page builder, or hand-built code. That crawl gets combined with other real records, whatever internal data the client makes available, so every number I put in front of you traces back to something actual rather than an estimate.
Once a new site is live, I do not stop crawling it. The same process runs periodically to catch problems early: pages that stopped loading, links that started breaking, structure that drifted from what it should be.
What it looks like in practice
You will not see the crawl itself. What you will see is a proposal or a report that cites specifics rather than generalities, this many pages, this particular gap, this exact structural issue, because those specifics came from a real pass through your site rather than a template.
If the site is later rebuilt as code with proper version control, future crawls become even more useful, because the state they report matches a state that is fully recorded and explainable, not a black box assembled on the fly.
Questions I get about this
- What does a crawl actually check?
- It walks through every page a site has, page by page, and records what is on each one, its content, its structure, whether it is indexed, how it performs, and how it links to other pages. It is the same basic process search engines use to find and understand your site.
- Do I need to do anything for a crawl to happen?
- No. It runs against the live site with no access needed to any account. I do it before I write a proposal so the numbers in it are real rather than estimated.
- Is a crawl the same as an SEO audit?
- A crawl is one input to an audit, the fact-gathering step. The audit is what I do with those facts afterward: deciding what is wrong, what matters, and what to fix first.
Want this set up properly for your business?
This is the kind of thing I build every week. Grab a time and we will talk through what fits.