Features - SilkCrawl SEO Spider

Flexible Scanning

Choose how you want to scan your website — three modes for different situations. Perfect for agencies juggling 20+ client sites with different structures.

Full website scan — enter your URL and SilkCrawl discovers every page automatically
Page list scan — upload a list of specific URLs you want to check
Sitemap scan — import your XML sitemap and verify every page listed in it

On-Page SEO Checks

Every page is automatically analyzed for the most important ranking factors. These are the exact issues Google uses to decide your ranking.

Page titles — missing, too short, too long, or duplicated across pages
Meta descriptions — missing or poorly optimized descriptions
Headings (H1-H6) — missing H1, multiple H1s, broken heading hierarchy
Canonical URLs — detect missing or incorrect canonical tags
Robots directives — find pages accidentally blocked from indexing
Hreflang tags — verify international/multilingual setup
Pagination — check rel=prev/next for paginated content
Breadcrumbs — detect breadcrumb markup for better search appearance
Word count — identify thin content pages that may not rank well

Link Analysis

Get a complete picture of your site's link structure — both internal and external. Broken links are low-hanging fruit — fix them fast, impress clients instantly.

Broken links — find all pages returning 404 errors
Internal link map — see how your pages connect to each other
External links — track every outbound link and where it goes
Nofollow links — identify which links pass authority and which don't
Link placement — see if links are in navigation, content, footer, or sidebar
Redirect chains — find links that go through multiple redirects
Inlink count — discover which pages receive the most internal links

Image Audit

Make sure your images are accessible and optimized for search engines. Missing alt text is one of the most common accessibility & SEO gaps on the web.

Missing alt text — images that screen readers and Google can't understand
Alt text too long — descriptions over 125 characters that may be truncated
Image formats — detect JPEG, PNG, WebP, AVIF, SVG usage across your site
Lazy loading — check if images use lazy loading for faster page speed

Structured Data & Social Tags

Verify that your pages appear correctly in search results and social media. Rich snippets can boost click-through rates by 20-30%.

Schema.org markup — detect JSON-LD and Microdata on every page
Schema types — see which types (Article, Product, FAQ, etc.) are used
Open Graph tags — check how pages appear when shared on Facebook and LinkedIn
Twitter Cards — verify Twitter/X preview appearance

Security Analysis PRO

Check your website's security configuration alongside your SEO audit. Show clients their SSL gaps — an easy upsell for web dev services.

SSL certificates — who issued it, when it expires, protocol version
HTTPS enforcement — detect if HSTS (strict transport security) is enabled
Mixed content — find insecure resources loaded on secure pages
Cookie security — check if cookies use Secure and HttpOnly flags

Custom Data Extraction PRO

Pull specific information from every page — prices, dates, phone numbers, or anything else. Great for competitor research and content audits at scale.

XPath queries — target specific HTML elements with precision
CSS selectors — extract data using familiar CSS targeting
Text pattern matching — find emails, phone numbers, or custom patterns across your site
Content search — search for specific words or phrases across all pages

Reports & Export

Share results with your team or clients in the format that works best. Agency owners: export client-ready reports in minutes, not hours.

Interactive HTML dashboard — visual report with charts, filters, and sortable tables
CSV export — open in Excel or Google Sheets for further analysis
Separate link reports — detailed internal links, external links, and image reports PRO
JSON export — structured data for integration with other tools PRO
XML sitemap generation — create a sitemap from your crawl results PRO

Performance & Settings

Fine-tune the scan to match your needs and server capacity. Smart rate limiting means you'll never crash a client's server.

Adjustable speed — control how fast SilkCrawl scans to avoid overloading your server
Page limits — set a maximum number of pages to scan
Depth limits — control how deep the spider follows links
URL exclusion — skip certain sections of your site (e.g., admin areas)
Custom headers — add authentication or special headers for protected sites
Robots.txt — optionally respect or ignore robots.txt rules