A high level of site usability is essential if its owner seeks to increase traffic, popularity, and visibility. However, it should be borne in mind that today web resources are visited not only by humans. Artificial intelligence also processes the information from sites to learn and generate relevant responses to user queries. In order for LLMs to be able to simply and conveniently obtain the data they need from the site, a special tool, LLMs.txt, was created. From this article, you will learn what it is, how to create it, and why you need it now.
What is LLMs.txt
LLMs.txt is a text-based file designed to configure LLM interactions with a website. This file is optimized for AI systems capable of understanding and analyzing information. Its main function is to act as a guide, suggesting which information from a web resource is most important.
The purpose of the LLMs.txt file is similar to the well-known sitemap.xml and robots.txt tools that have helped search engines find and index pages faster and correctly for decades. It gives LLMs only the information they need from a website in a clearly organized format that is as easy to process and analyze as possible.
A logical question arises: why was it necessary to introduce an additional file when, for example, sitemap.xml already contains a list of all pages of a web resource? The fact is that such a list does not give AI a complete understanding of what exactly is on these pages. The robots.txt file is used to control search engine access to a site. Of course, they will not help AI analyze the materials posted on web pages. Instead, the LLMs.txt file contains structured data and instructions for analysis by AI (without unnecessary elements such as JavaScript, HTML styles, or advertising).
There are two types of LLMs.txt documents:
- /llms.txt – provides a compact, simplified package of site data.
- /llms-full.txt – contains complete information about all the content of the web resource.
This file is primarily used for websites. However, it can also be useful for other resources. It allows AI systems to more easily access and process corporate materials (security policies, technical specifications, license agreements, etc.), API documentation, and even cloud storage.
Who Invented LLMs.txt and Why
The idea of LLMs.txt first emerged publicly in September 2024. Its author is Jeremy Howard, one of the founders of the IT lab Answer.AI. In his opinion, this text file will be a solution to an important problem that currently hinders all LLMs.
The root of the problem lies in the limited context window, which any AI model, even the most advanced one, suffers from. This makes it difficult for LLMs to process large amounts of web content and documents. When a language model analyzes a web page, everything gets into its context window, including many useless elements: scripts, HTML styles, navigation, etc. The LLMs.txt file can save the AI system from such inconveniences, directing it exclusively to the content it needs, removing all unnecessary components. It uses a format optimized for machine analysis.
In addition, Jeremy Howard proposed another method of optimizing web pages for AI. It consists of creating alternative adapted versions of them. They should be cleaned of all unnecessary elements and contain only useful information for LLMs. The alternative page should be placed at the same URL as the original, but with .md appended to the end. This is how Markdown documentation is designated. For example, the URL of an alternative version of the site's homepage optimized for AI would be index.html.md.
How to Create LLMs.txt
The LLMs.txt file will work correctly if no errors are made during its creation. The main rule is that it is written only in Markdown. The file should maintain a consistent formatting style. It should not be a dump of chaotic lists and disorganized paragraphs that have no logical connection.
Step-by-step algorithm:
- In the first line, write the title in H1 (#) format. Usually, this is the name of your web resource or project.
- In the second line, create a short info block and format it as a quote (>). Give a brief description of the topic of your site and its content. Try to keep it to 1–3 sentences.
- In the third and subsequent lines, enter logical block sections (“Basic documentation,” “Product catalog,” “User manual,” etc.). The name format of each such block is H2 (##).
- Add a description and URL to each logical block. For example, the “Product Catalog” block should be accompanied by a link to this page on your site and a short description of the content contained in this section. Blocks should be created clearly according to the following scheme:
[Block name](URL):Short description. - If necessary, supplement the file with additional sections (“Examples of Use,” “Privacy Policy,” etc.).
LLMs.txt example:
If you are unsure whether you have written everything correctly, use the resource llmstxt.directory. There you can view files that have been created and made public by a number of companies.
The file can be created manually or automatically. Which method is better? By creating a document manually, you decide what to put in it and have full control over the process. Using the LLMs.txt generator will allow you to do this much faster and avoid errors.
Toolkit for creating LLMs.txt:
- llmstxt.firecrawl.dev – an online generator for heavyweight platforms with hundreds or thousands of pages of content.
- llmstxtgenerator.org – an online generator of basic llms.txt for lightweight sites with a small number of pages.
- Mintlify – a platform for automatic generation and placement of llms.txt and llms-full.txt.
- LLMs.txt for WP – a plugin for WordPress with support for the .md format for auto-generation of llms.txt. It works only for sites built using this platform.
- The GitHub repository AnswerDotAI/llms-txt provides tools for automatically generating an llms.txt file for your website. You can also view sample llms.txt files there to understand what the content and structure of this document should be.
The finished LLMs.txt file should be placed in the root folder of your website. Example: www.mycompany.com/llms.txt
After that, check if it is available. To do this, open any browser, type www.mycompany.com/llms.txt and press Enter. If the file does not open, check that it was downloaded correctly, access rights, and review the server settings.
The Impact of LLMs.txt on Website Promotion
By equipping your website with the LLMs.txt file, you will receive important benefits for promoting the resource:
- Generative AI chatbots (ChatGPT, Gemini, Copilot, Claude, etc.) may be more likely to recommend your site as a source of information in their responses to user queries.
- With a structured guide, LLMs navigate content faster and better. Accordingly, they will better explain it to users.
- If AI systems often mark your site as a source of information, the LLMs.txt file will allow them to provide reliable and accurate answers as possible without having to guess anything themselves.
- If AI regularly uses your site's content in its responses with a link to it, it will bring more interested visitors to your resource. Another significant advantage is the increase in the level of trust of users in it.
The spread and implementation of generative AI models is happening so quickly that it is simply impossible to ignore. This is especially clearly observed in digital marketing, in particular in search engine optimization. Classic SEO is no longer enough today. The clarity of the resource for artificial intelligence now matters no less than traditional SEO methods and techniques. LLMs.txt is a new tool that adapts your site to successfully interact with large language models, makes it AI-friendly, and increases the efficiency of its indexing in AI-powered search engines.
SaveMyLeads is a simple and effective service that will help you automate routine tasks and optimize business processes. Stop wasting time uploading leads from Facebook manually – you can do it automatically, saving a lot of time and money. Eliminate routine from workflows and achieve more with minimal investment of money, effort and human resources.