LinkStorm uses a powerful web crawler to analyze your site’s internal linking structure, gathering essential data to help you optimize your links for SEO. Understanding how the crawler works will help you interpret reports and make better internal linking decisions.
How LinkStorm Crawls Your Site
Discovering URLs
LinkStorm finds pages on your site through:
Your Sitemap – If an XML sitemap is available, LinkStorm uses it as the primary source for URL discovery.
Internal Links – The crawler follows links on your pages to discover additional content.
💡 Best practice: For the most accurate analysis, ensure your sitemap is up to date and contains all indexable pages.
Crawling Behavior
When crawling, LinkStorm:
Renders JavaScript – This allows the crawler to fully load dynamic content, making it compatible with any website or CMS, including JavaScript-heavy frameworks.
Honors meta directives – Pages with
noindexwon’t be considered indexable.Detects Canonical Pages – If a page has a canonical tag pointing elsewhere, LinkStorm treats it as non-canonical.
Limits Crawling to the Specified Scope – If a subdomain or path is entered, LinkStorm will only crawl pages within that specific subdomain or path (Learn how to organize your websites and projects).
Adjusts crawl speed for slow sites – If your site loads slowly, LinkStorm reduces resources allocated to crawling it to prevent bottlenecks.
💡 Best practice: Optimize site performance to improve crawl efficiency, especially for large websites.
How Often LinkStorm Crawls Your Site
LinkStorm now performs:
A full crawl when a new project is created – This gathers initial data.
An automatic full recrawl every 30 days – This keeps your data fresh without requiring manual action.
What Data LinkStorm Collects
During a crawl, LinkStorm gathers:
Page URLs – List of all discovered pages.
Internal Links – Connections between pages.
External Links – Outbound links pointing to other sites.
Anchor Texts – The words used in internal and external links.
Indexability & Canonical Status – Determines whether a page is indexable and canonical.
Issues – Broken links, redirects, and nofollow attribute.
💡 Internal link opportunities are computed once the crawl is completed.
Next Steps
Now that you understand how LinkStorm crawls your website, you can:
Review the Pages Report to analyze your indexed and non-indexed pages.
Check the Issues Report to fix broken links or redirects.
Use the Opportunities Report to implement AI-recommended internal links.