CrawlTrace AI - AI Crawlability & Schema Auditor
Check whether AI search crawlers can reach and understand your website. CrawlTrace AI audits robots.txt, llms.txt, sitemaps, page markup and JSON-LD structured data, then returns severity-ranked findings with evidence, ready-to-paste robots.txt fixes and JSON-LD templates, and a 30-day action plan. Checks run in a deterministic script, and anything it could not measure is stated. Built for SEO professionals, agencies and site owners. It does not promise rankings or AI citations.
#Engineering#Research#Marketing
Rating
More ratings needed
Sold
0
How to use
Download
CrawlTrace AI: AI Crawlability & Schema Auditor
Can AI search crawlers reach your website, and can they understand it?
CrawlTrace AI answers that with evidence. It reads your public crawler rules, discovery files and page HTML, measures them with a deterministic script, and returns a prioritized fix list you can act on today.
It is the technical companion to GEOTrace AI: GEOTrace measures what AI answers say about your brand, and CrawlTrace checks the site-side barriers that can get in the way.
What you get
- Crawler access table: allowed, partial or blocked status for Googlebot, Bingbot and the OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Amazon crawlers, with the exact robots.txt rule responsible
- Page checks: HTTP status, noindex and snippet restrictions, pages that depend on JavaScript for their content, title, H1, canonical, language, image alt text
- Structured data checks: JSON-LD validity, schema.org context, required and recommended properties for Organization, WebSite, Article, Product, Person, FAQPage, BreadcrumbList, VideoObject and LocalBusiness
- Discovery checks: sitemap coverage of your key pages, lastmod dates, llms.txt presence and format, Sitemap directive in robots.txt
- Severity-ranked findings: each one has the evidence that triggered it and a fix
- Ready-to-apply patches: robots.txt corrections and JSON-LD templates, with placeholders marked wherever a fact must come from you
- 30-day action plan and a machine-readable JSON report
How to use it
Give the agent:
- Your domain, for example
example.com - Up to 10 priority page URLs (optional; if you give none, the agent audits your homepage and a few pages it finds)
- Optionally, attach files: robots.txt, sitemap.xml or page HTML source
You do not need to share passwords, API keys, cookies or analytics access.
Example findings
This is from the bundled sample audit of a fictional site (example.com), not a real customer.
| Severity | Target | Evidence | Fix |
|---|---|---|---|
| High | /blog/ai-search-guide | JSON-LD block is not valid JSON: illegal trailing comma | Fix the JSON syntax; invalid blocks are ignored |
| Medium | PerplexityBot | Partially blocked by Disallow: /blog/ |
Remove or narrow the rule if you want inclusion in Perplexity answers |
| Medium | /products/widget | nosnippet found |
Remove it unless you intend to prevent excerpts |
| Medium | /app | 0 visible words in raw HTML, 3 script tags | Server-render the main content |
| Info | GPTBot, ClaudeBot | Blocked | No action needed if intentional: training crawlers are your policy choice and are not scored |
Why you can trust the numbers
- Deterministic: a script with no network access runs the checks, so the same input always gives the same result, and every finding cites its evidence.
- Honest scoring: scores are checks passed out of checks applicable for your sample. They are not industry benchmarks.
- No invented content: the agent never fabricates a fetched file, and it never makes up authors, dates, logos or profile URLs in JSON-LD patches.
- Clear limits: anything it could not measure is listed under "Not measured".
Limits
- It reads robots.txt rules, not live crawler behavior. It cannot see firewall, CDN or bot-protection blocks, server logs, or whether a crawler is genuine.
- Page markup and structured-data checks need the page's raw HTML. If only rendered text is available, those checks are reported as not measured and you can attach the page source.
- It does not render JavaScript and does not observe AI answers. To see what AI answers say about you, use a citation audit such as GEOTrace AI.
- llms.txt is a proposed convention. No major AI provider has confirmed that it affects citations, so it is rated low priority.
- It does not guarantee rankings, citations, traffic or revenue.
Data handling
The agent requests only public pages on your domain, using the runtime's own fetch tool, and states which tool it used in the report. The analysis script runs offline. Pages you did not ask about, and other domains, are not fetched.
Version
1.0.1
