LinkStorm uses a powerful web crawler to analyze your site’s internal linking structure, gathering essential data to help you optimize your links for SEO. Understanding how the crawler works will help you interpret reports and make better internal linking decisions.
How LinkStorm Crawls Your Site
Discovering URLs
LinkStorm finds pages on your site through:
Your Sitemap – If XML sitemap processing is enabled, LinkStorm will use it as the primary source for URL discovery.
Internal Links – The crawler follows links on your pages to discover additional content.
💡 Best practice: For the most accurate analysis, ensure your sitemap is up to date and contains all indexable pages.
Crawling Behavior
When crawling, LinkStorm:
Renders JavaScript – This allows the crawler to fully load dynamic content, making it compatible with any website or CMS, including JavaScript-heavy frameworks.
Honors meta directives – Pages with
noindexwon’t be considered indexable.Detects Canonical Pages – If a page has a canonical tag pointing elsewhere, LinkStorm treats it as non-canonical.
Limits Crawling to the Specified Scope – If a subdomain or path is entered, LinkStorm will only crawl pages within that specific subdomain or path (Learn how to organize your websites and projects).
Adjusts Crawl Speed for Slow Sites – If a website loads very slowly or is poorly optimized, LinkStorm may reduce the resources allocated to crawling it. This helps ensure that other sites are not affected by long processing times.
💡 Best practice: To avoid slow crawl speeds, ensure your site is optimized for performance, especially for large websites.
How Often LinkStorm Crawls Your Site
LinkStorm crawls your site:
When a new project is created – A full crawl is performed to gather initial data.
When you trigger a manual crawl – You can start a new crawl from the dashboard (this consumes credits).
Weekly for Sitemap Processing – If enabled, LinkStorm will check the XML sitemap once per week for new or updated content.
💡 To manually trigger a crawl, go to the Dashboard and click "Reset and recrawl now"
What Data LinkStorm Collects
During a crawl, LinkStorm gathers:
Page URLs – List of all discovered pages.
Internal Links – Connections between pages.
External Links – Outbound links pointing to other sites.
Anchor Texts – The words used in internal and external links.
Indexability & Canonical Status – Determines whether a page is indexable and canonical.
Issues – Broken links, redirects, and nofollow attribute.
💡 Internal link opportunities are computed once the crawl is completed.
Managing Your Crawls
Controlling What Gets Crawled
To refine your crawls, you can:
Enable or disable sitemap processing in your dashboard.
Define a subdomain or path when adding a website to restrict crawling to specific sections.
When to Run a Manual Crawl
Consider running a new crawl if:
You’ve added or removed major site sections.
You’ve changed the internal linking structure.
You want the latest data before optimizing links.
Next Steps
Now that you understand how LinkStorm crawls your website, you can:
Review the Pages Report to analyze your indexed and non-indexed pages.
Check the Issues Report to fix broken links or redirects.
Use the Opportunities Report to implement AI-recommended internal links.