Make use of mkdocs-llmstxt
#1427
Conversation
PR Change SummaryIntroduced support for the llms.txt format in the PydanticAI documentation.
Modified Files
How can I customize these reviews?Check out the Hyperlint AI Reviewer docs for more information on how to customize the review. If you just want to ignore it on this PR, you can add the Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add |
Viicos
force-pushed
the
mkdocsllmstxt
branch
2 times, most recently
from
7384387 to
1df21f8
Compare
Docs Preview
|
mkdocs.yml
Outdated
| sections: | ||
| Concepts documentation: | ||
| - agents.md | ||
| - models.md | ||
| - dependencies.md | ||
| - tools.md | ||
| - common-tools.md | ||
| - results.md | ||
| - message-history.md | ||
| - multi-agent-applications.md | ||
| Graphs: | ||
| - graph.md | ||
| Evals: | ||
| - evals.md | ||
| MCP: | ||
| - mcp/*.md | ||
| Optional: | ||
| - testing.md | ||
| - cli.md | ||
| - logfire.md | ||
| - examples/*.md |
There was a problem hiding this comment.
can't we opt-out for pages instead?
There was a problem hiding this comment.
Not as of today, but you can open a feature request at https://github.com/pawamoy/mkdocs-llmstxt
There was a problem hiding this comment.
The thing is the llms.txt format is defined as being "flat" (see https://llmstxt.org/#format), so it can't really follow the documentation structure that may be nested under multiple levels.
There was a problem hiding this comment.
I'm not going to remember to update this when I add a new page to the docs. I need something that can keep the docs up-to-date.
There was a problem hiding this comment.
Can this section be generated dynamically as part of the deployment pipeline in github actions, possibly reusing the existing nav, specifying opt-outs if needed? The default behaviour would then be to include all documentation.
There was a problem hiding this comment.
It would require dynamically generating the MkDocs config file in some way, and as I mentioned above we can't blindly follow the documentation layout as the llms.txt format can't be nested.
Imo the llms.txt format should be the one evolving instead. This flat layout limitation doesn't really make sense and most probably can be removed. It would then be easier for us to follow the documentation structure and exclude some pages if wanted.
| ## llms.txt | ||
|
|
||
| The PydanticAI documentation is available in the [llms.txt](https://llmstxt.org/) format. | ||
| This format is defined in Markdown and suited for large language models. | ||
|
|
||
| Two formats are available: | ||
|
|
||
| - [llms.txt](https://ai.pydantic.dev/llms.txt): a file containing a brief description | ||
| of the project, along with links to the different sections of the documentation. The structure | ||
| of this file is described in details [here](https://llmstxt.org/#format). | ||
| - [llms-full.txt](https://ai.pydantic.dev/llms-full.txt): Similar to the `llms.txt` file, | ||
| but every link content is included. Note that this file may be too large for some LLMs. | ||
|
|
||
| As of today, these files *cannot* be natively leveraged by LLM frameworks or IDEs. Alternatively, | ||
| an [MCP server](https://modelcontextprotocol.io/) can be implemented to properly parse the `llms.txt` | ||
| file. | ||
|
|
||
|
|
There was a problem hiding this comment.
Do we need to document this? Isn't it already useful to have it discoverable?
There was a problem hiding this comment.
We decided to go this way a when discussing it in a call: have it documented with a small section on Pydantic, pAI and logfire
There was a problem hiding this comment.
We can also leave it for a while for users to be aware of it and remove it at a later point.
|
closed and opened to trigger the pipeline |
Fixes #1028.