Crawling
Multiple scan modes for different situations — from quick homepage crawls to full sitemap verification. Perfect for agencies juggling 20+ client sites with different structures.
- Full website crawl — enter your URL and discover every page automatically
- Start from homepage (root domain) — default for all crawls
- Start from any URL (deep-start) (Pro)
- Crawl from URL list (.txt file) (Pro)
- Crawl from XML sitemap (Pro)
- Scheduled recurring crawls — automate audits on a schedule
- Compare two crawls — diff results to track changes over time
- Generate reports from saved crawl data — no re-crawling needed
- Page limit — Free: 200 pages, Pro: unlimited
On-Page SEO Analysis
Every page is automatically analyzed for the most important ranking factors. These are the exact issues Google uses to decide your ranking.
- Page titles — missing, too short, too long, or duplicated across pages
- Meta descriptions — missing or poorly optimized descriptions
- Headings (H1–H6) — missing H1, multiple H1s, broken hierarchy
- Canonical URLs — self-canonical, cross-domain detection
- Meta robots / X-Robots-Tag — find pages accidentally blocked from indexing
- Word count — identify thin content pages that may not rank well
- Pagination — rel=next / rel=prev verification
- Breadcrumbs — breadcrumb markup detection
Link Analysis
Get a complete picture of your site's link structure — internal and external. Broken links are low-hanging fruit — fix them fast, impress clients instantly.
- Internal links — URL, anchor text, nofollow detection
- Link position on page — nav, header, footer, main, aside, article
- External links — URL, anchor text, nofollow detection
- External link verification — check if external links return errors
- Redirect chain detection — find multi-hop redirects
Image Audit
Make sure your images are accessible and optimized for search engines. Missing alt text is one of the most common accessibility & SEO gaps on the web.
- Missing alt text — images that screen readers and Google can't understand
- Alt text too long — descriptions over 125 characters that may be truncated
- Image formats — detect JPEG, PNG, WebP, GIF, SVG usage across your site
- Lazy loading detection — check if images use lazy loading for faster page speed
Structured Data
Detect and validate structured data markup across your entire site. Rich snippets can boost click-through rates by 20–30%.
- JSON-LD, Microdata, and RDFa detection on every page
- Schema.org type identification — see which types are used
- Google validation — check required and recommended properties
- 14+ supported types: Article, Product, FAQPage, Recipe, Event, LocalBusiness, Organization, HowTo, VideoObject, BreadcrumbList, BlogPosting, NewsArticle, Offer, AggregateOffer
Security Analysis
Check your website's security configuration alongside your SEO audit. Show clients their SSL gaps — an easy upsell for web dev services.
- SSL/TLS certificates — issuer, expiry date, protocol version
- HSTS enforcement — strict transport security detection
- Mixed content — insecure HTTP resources on HTTPS pages
- Cookie security — Secure and HttpOnly flag verification
- Exposed files scanning — detect .env, .sql, .git, .log, .backup, archives
- SSRF protection — built-in private IP blocking during crawls
Advanced Crawling
Fine-tune the crawl engine for any scenario — SPAs, authenticated sites, or staging environments. Smart rate limiting means you'll never crash a client's server.
- JavaScript rendering — Playwright / headless Chromium for SPAs
- CSS/JS/font resource scanning — analyze all static resources
- Proxy support — HTTP and SOCKS5
- Basic Auth — user:password for protected sites
- Custom cookies and HTTP headers
- Custom User-Agent — impersonate any browser or bot
- Robots.txt — respect or ignore directives
- URL exclusion patterns — regex-based path filtering
- Stay-in-path / include subdomains
- SSL verification toggle — skip for staging sites
Custom Data Extraction
Pull specific information from every page — prices, dates, phone numbers, or anything else. Great for competitor research and content audits at scale.
- XPath extraction — named expressions targeting specific HTML elements
- CSS selector extraction — familiar CSS targeting for data
- Regex extraction — match text patterns across all pages
- HTML content search — find specific words or phrases site-wide
Reports & Exports
Share results with your team or clients in the format that works best. Agency owners: export client-ready reports in minutes, not hours.
- CSV main report — 45+ columns plus custom extraction columns
- Specialized CSVs — internal links, external links, images, inlinks, resources, redirect chains, canonicals, hreflang, exposed files
- JSON export — full structured data for tool integration
- Interactive HTML Dashboard — charts, filters, sortable tables
- Crawl Map HTML — visual site structure visualization
- XML Sitemap generation — from crawl results
- Save/reload crawl data — resume analysis without re-crawling
Three Interfaces
Use SilkCrawl the way that fits your workflow — graphical, terminal, or command-line.
- GUI (PySide6) — full graphical interface, cross-platform
- CLI — command-line for scripting and automation
- TUI (Textual) — interactive terminal UI with panels
- 13 data tabs with filters, sorting, and pagination
- Detail panel — click any row for full page analysis
- SEO Help — contextual tips for every metric
- Export dialog — choose format and data to export
- Compare dialog — select two crawls to diff
Pro license: $99 one-time · 14-day money-back guarantee · no subscription
Social Tags & Hreflang
Verify that your pages appear correctly when shared on social media and verify your multilingual setup. Wrong OG tags = ugly link previews = fewer clicks.