Pillar 1 · AI Search, GEO & AEO
Machine-Readable Web Standards: llms.txt, Crawlability, and Schema
Updated 2026-10-24 · SYS CRED · Prince George, BC
Direct answer
AI crawlers and search bots need clean access and accurate definitions. Publish JSON-LD that matches the page (Organization, Service, TechArticle, FAQPage), keep robots.txt intentional, and consider an /llms.txt file that points agents at your best facts — without blocking the crawlers you want citing you.
llms.txt and markdown for agents
llms.txt is an emerging convention: a plain-text map of URLs and short summaries for AI agents. It does not replace robots.txt or sitemap.xml. It highlights the pages you want models to prioritize.
Keep it short and factual. Link to about, services, locations, and main guides. Do not stuff keywords.
JSON-LD that matches the visible page
Use Organization, LocalBusiness, or ProfessionalService with consistent legalName, name, address locality, email, areaServed, and sameAs. Attach Service nodes to real service URLs. Use TechArticle on technical blogs and FAQPage only where FAQs appear on the page.
Do not add AggregateRating markup you did not earn. Fake review schema risks Google penalties and buyer trust.
- One @id for the organization
- Canonical URLs in every node
- Same contact facts as the footer NAP
Crawler budget and AI bots
GPTBot, ClaudeBot, and similar bots follow robots.txt when it is set correctly. Allow bots you want citing you. Block scrapers only if that is a deliberate policy.
On larger sites, soft 404s, parameter duplicates, and filter URLs waste crawl time. Keep sitemaps focused and use permanent redirects for retired URLs.
Frequently asked questions
Is llms.txt required?+–
Not by any major engine today. It is a useful clarity layer, especially if you offer GEO services.
Should we block all AI bots?+–
Usually no if you want citations. Blocking can reduce scraping but also reduces discovery. Decide per bot.
Does schema alone rank us?+–
No. Schema clarifies content; quality and links still matter. Wrong schema can hurt.