Making content readable for LLMs and external agents
llms.txt, Markdown versions of pages, structured data and JSON output give agents a representation of the site they can use directly.
| Capability | Origin and component | What it provides |
|---|---|---|
| Provide llms.txt and agent skills from configuration | Netresearch extension nr-wellknown (beta) |
At deploy time, writes /llms.txt as a static file from a file or inline text in the site configuration. When skills are configured, it also writes /.well-known/agent-skills.json. Limits: There is one file per docroot: in a multi-site instance, the configuration of the first site applies. The agent-skills.json format is not a widely supported standard. Sources: nr-wellknown on GitHub |
| Generate llms.txt from the page tree | Community extension web-vision/ai-llms-txt by web-vision |
Builds llms.txt from the page tree, with configurable depth and page-type exclusions. It leaves out hidden, access-restricted and no-index pages. It also serves pages as Markdown and exports them through a CLI command. Sources: ai-llms-txt manual |
| llms.txt with Markdown endpoints | Community extension rtfirst/llms-txt by Roland Tfirst |
Generates llms.txt with Markdown access to page content and can protect these endpoints with an API key. Sources: llms-txt manual |
| Pages as Markdown | Community extensionNetresearch integration b13/ai-bots-love-markdown by b13, used in the Netresearch demo |
Serves a page as Markdown when the request sends Accept: text/markdown or uses a URL suffix. The Markdown carries the page metadata as front matter, and the HTML page links to it with rel="alternate". Sources: ai-bots-love-markdown on GitHub |
| Describe content semantically | Community extension brotkrueml/schema by Chris Müller |
Adds schema.org structured data to pages as JSON-LD, so content types and relations are machine-readable. Sources: schema manual |
| Pages and content as JSON | Community extension friendsoftypo3/headless by the TYPO3 Headless initiative |
Renders pages, content elements, menus and metadata as JSON, with multilingual handling and workspace preview. Sources: headless manual |
| Deliver content by context | Netresearch extension contexts, combinable with Cowriter |
Shows or hides pages and content by configurable contexts. One context matches requests from known AI crawlers. Cowriter can create audience variants that an editor can bind to such a context. Limits: The client sets the User-Agent header. The AI-crawler context therefore classifies requests and does not authenticate them. Sources: contexts on GitHub · Cowriter on GitHub |
- The Netresearch TYPO3 demo serves a hand-curated llms.txt directly from nginx and offers every page as Markdown via b13/ai-bots-love-markdown. It does not use nr-wellknown.
- llms.txt is a proposal for giving agents a curated overview of a site. It does not guarantee that AI systems use or rank the content. Markdown and structured data are representations that agents can use directly.