Start from your homepage and sitemap, follow every internal link, render JavaScript when you need to, and get a full page inventory you can filter and export.
An SEO crawler visits your pages the way a search engine bot does, following links from page to page and recording what it finds. The BigRank crawler collects status codes, titles, headings, links and images for every URL, then keeps each run as a snapshot.
Each crawl starts from your homepage and sitemap and follows internal links breadth-first within your URL and depth limits. Scope, user-agent and outbound IP stay the same on every run, so snapshots can be compared.
An asynchronous HTTP crawler with HTTP/2 and connection pooling fetches many pages at once, then parses the HTML quickly.
For sites that load content with JavaScript, a browser renders each page before it is checked.
The crawl starts at the homepage and sitemap, follows internal links up to your limits, and skips subdomains unless you include them.
Add URL patterns to keep the crawl on the sections you care about and away from the ones you don't.
The crawler respects robots.txt, applies a per-host rate limit, identifies itself with its own user-agent and a fixed outbound IP, and retries with backoff.
Follow each crawl from Waiting to Crawling, Checks and Results, with live counters for pages analyzed and responses.
Every crawl is kept as a snapshot, and a date selector lets you switch between past runs.
Download any table, with its filters applied, as CSV or XLSX.
See every URL with its status, response time, word count, issue tags, link counts, image counts and depth in one filterable table.
Set the maximum number of URLs and crawl depth, add include and exclude patterns, and decide whether subdomains belong in the crawl.
Run a crawl by hand with the Analyze button, or let scheduled recrawls keep the data fresh. Weekly is the default, with daily runs on higher plans.
SEO MCP
Ask your assistant how many pages the latest crawl analyzed, which URLs returned an error status, or how the last crawl compares with the one before.
A crawl is only as good as its scope. Set the limits and access rules before the first run, then keep them fixed so every snapshot measures the same site.
Choose the maximum URLs, depth and whether your content needs JavaScript rendering.
Allowlist the crawler's fixed IP and add the robots.txt snippet for its user-agent.
Follow live progress while the crawl moves through each stage.
Narrow the page list to what matters and export it for your team.
Reach for the crawler when you need raw page data rather than a verdict: a full URL inventory, proof that rendered content is visible, one section of a large site, or a list for developers.
Get a full list of URLs, titles and statuses before you plan a content or technical project.
Confirm that content rendered by JavaScript is visible once pages are loaded in a browser.
Include or exclude URL patterns to crawl one part of a large site.
Export filtered page lists so engineers can work from exact URLs.
Use SEO Crawler for the Pages table with every URL, status and on-page detail, Broken Link Checker for the Links table narrowed to failures, and Image SEO Analysis for the Images table with alt, size and format checks.
Crawl from your homepage and sitemap with your own limits, render JavaScript when needed, and filter or export every page found.
List each link once by target URL with its status and source pages, then filter to broken targets, redirects and flagged attributes.
Review every image the crawler finds, with missing alt text, missing dimensions, legacy formats and files over 200 KB.
Keep going
The crawl feeds the rest of the product. Read the issues it found, check how Google indexed the same pages, and follow the health score as you fix them.
Answers about page limits per plan, robots.txt, firewall access, whether old crawls are kept, crawling part of a site and how the crawler treats a slow server.
Up to 5,000 pages per project on Pro, 10,000 on Max and 50,000 on Ultra.
Yes. It respects robots.txt, with an override for verified owners, and limits requests per host.
Project setup shows the crawler's fixed outbound IP to allowlist, plus a robots.txt snippet for its user-agent.
Yes. Each crawl is saved as a snapshot you can open from the date selector.
Yes. Add include and exclude URL patterns, and set the maximum URLs and depth for the project.
The crawler applies a per-host rate limit, retries with backoff and stops waiting after a timeout, so a slow server isn't flooded with requests.
Plans start at $19/month with a 7-day free trial. Every plan includes the site audit, Search Console insights and white-label reports.