Skip to main content
All CollectionsGetting Started
How LinkStorm Crawls Your Website
How LinkStorm Crawls Your Website

Learn how LinkStorm crawls your website, discovers pages, analyzes internal links, renders JavaScript, and processes sitemaps to optimize your internal linking strategy.

Updated yesterday

LinkStorm uses a powerful web crawler to analyze your site’s internal linking structure, gathering essential data to help you optimize your links for SEO. Understanding how the crawler works will help you interpret reports and make better internal linking decisions.

How LinkStorm Crawls Your Site

Discovering URLs

LinkStorm finds pages on your site through:

  • Your Sitemap – If XML sitemap processing is enabled, LinkStorm will use it as the primary source for URL discovery.

  • Internal Links – The crawler follows links on your pages to discover additional content.

💡 Best practice: For the most accurate analysis, ensure your sitemap is up to date and contains all indexable pages.

Crawling Behavior

When crawling, LinkStorm:

  • Renders JavaScript – This allows the crawler to fully load dynamic content, making it compatible with any website or CMS, including JavaScript-heavy frameworks.

  • Honors meta directives – Pages with noindex won’t be considered indexable.

  • Detects Canonical Pages – If a page has a canonical tag pointing elsewhere, LinkStorm treats it as non-canonical.

  • Limits Crawling to the Specified Scope – If a subdomain or path is entered, LinkStorm will only crawl pages within that specific subdomain or path (Learn how to organize your websites and projects).

How Often LinkStorm Crawls Your Site

LinkStorm crawls your site:

  • When a new project is created – A full crawl is performed to gather initial data.

  • When you trigger a manual crawl – You can start a new crawl from the dashboard (this consumes credits).

  • Weekly for Sitemap Processing – If enabled, LinkStorm will check the XML sitemap once per week for new or updated content.

💡 To manually trigger a crawl, go to the Dashboard and click "Reset and recrawl now"

What Data LinkStorm Collects

During a crawl, LinkStorm gathers:

  • Page URLs – List of all discovered pages.

  • Internal Links – Connections between pages.

  • External Links – Outbound links pointing to other sites.

  • Anchor Texts – The words used in internal and external links.

  • Indexability & Canonical Status – Determines whether a page is indexed, non-indexable, or non-canonical.

  • Issues – Broken links, redirects, and nofollow tags.

💡 Internal link opportunities are computed once the crawl is completed.

Managing Your Crawls

Controlling What Gets Crawled

To refine your crawls, you can:

  • Enable or disable sitemap processing in your dashboard.

  • Define a subdomain or path when adding a website to restrict crawling to specific sections.

When to Run a Manual Crawl

Consider running a new crawl if:

  • You’ve added or removed major site sections.

  • You’ve changed the internal linking structure.

  • You want the latest data before optimizing links.

Learn how to trigger a manual crawl.

Next Steps

Now that you understand how LinkStorm crawls your website, you can:

Did this answer your question?