How to Create and Set Up an llms.txt File

Step by Step Guide: How to Create and Implement an llms.txt File

Today, we’re diving into something that’s really making waves in the tech world and changing the way AI interacts with the internet.

We’ve looked into a bunch of articles about llms.txt, and our mission today is to clear things up: what they are, why everyone’s talking about them, and – most importantly – what impact they can have on you. We’re also getting into how to actually implement llms.txt, along with some interesting comparisons to familiar web standards like robots.txt. It really gives you a solid snapshot of something fresh and new.

This article is valuable because it’s like getting a head start on understanding something that might become a basic part of the web as AI models get more intertwined with how we search for and use information. And yeah, it’s all happening pretty quickly. Grasping llms.txt could give you a real advantage.

What the H**k is llms.txt?

Think of llms.txt as the younger cousin of robots.txt. It is a simple Markdown-formatted text file you stash in your website’s root directory (yep, right next to robots.txt). But instead of bossing around search engine crawlers, it’s designed to guide large language models (LLMs) (like ChatGPT, Claude, or Gemini) on how to process your content.

Proposed in 2024 by Jeremy Howard from Answer.ai, llms.txt acts like a CliffsNotes version of your site for AI. It serves 3 primary functions:

Content Prioritization: Highlights your most important pages and resources
Context Provision: Explains your site’s purpose and content relationships
Access Control: Specifies which content should be ignored or treated carefully

In simple words, llms.txt is a handy guide for LLMs during a stage called the inference phase when they’re actually doing their thinking. So, when the AI is coming up with an answer or using what it knows to respond to your question, that llm.txt file gives it detailed instructions.

It shows the AI which info on the website to focus on and what to pay attention to. The whole point is to provide LLMs with content that’s formatted and organized just for them. It’s all about concise, expert-level info that’s super easy to find.

Why Markdown Format?

You’ve probably noticed preferred format of the llms.txt file is Markdown. So, why is that? Well, it’s pretty simple—Markdown is just super easy to read for both people and machines. It lays out information in a clear and structured way without all the fuss of full-on HTML coding. LLMs don’t have to deal with all those tags and scripts, which is a relief. That simplicity means LLMs can quickly parse it and understand the content with way less confusion, making everything process more efficiently.

Why Does This Matter for You?

You might be thinking, “Sure, this llms.txt file could benefit the folks behind the Large Language Models, but they’re already rolling in dough. So why should I, as a website owner or content creator, even care about it?

Story Time!

In the early days of search engines, lots of media companies tried to shut out Google’s indexing bots. But eventually, they realized they had to be a part of the search engine game.

Now we’re seeing a similar situation again. There’s a concern that proprietary data could be misused to train LLMs. But can you really afford to miss out on the opportunity? LLM indexing bots might show up on your site.

My take on this whole thing is, if you can’t beat LLMs, why not team up with them? That’s where llms.txt comes in, and it’s pretty important for businesses, brands, content creators—basically anyone with a website. It gives a bit of control over how AI models can access and use their content, which hasn’t really been a thing until now.

Some experiments show that using llms.txt could help models cite your content more accurately, completely, and in a way that makes more sense. That’s a total game-changer for keeping your online reputation in check.

It’s like getting a say in how your info pops up in AI responses, which is a big deal! While it’s not guaranteed to boost SEO like traditional methods, but backed by a decent amount of evidence so far.

Pro Tip

Imagine an AI summarizing a product review page. With a well-structured llms.txt, it could be instructed to prioritize verified customer testimonials and downplay any potentially biased comments from forums.

This way, the AI directly shapes its output, helping to protect your brand image when someone’s researching your product.

Guiding the AI to the Right Pages

Site owners can also guide AI in prioritizing content, telling it what bits to highlight. For instance, you could even indicate that an article is outdated so the AI understands not to treat it as the latest info.

Plus, if there’s proprietary content or behind a paywall, llms.txt can signal to AI models that it should stay private unless the proper permissions are in place.

Performance Perks

There’s also a perk in terms of server load. If LLMs can get what they need from llms.txt efficiently, they won’t need to crawl the entire site, which is a win for website performance!

Understanding llms.txt vs. llms-full.txt: What’s the Difference?

I promise this is the last curveball I’m throwing in this article. Before continuing, you should know that llms.txt comes in two flavors. The first one you probably already know is llms.txt, and the second one is llms-full.txt.

The difference between them is pretty interesting. Let me break it down for you!

Quick vs. Comprehensive

A helpful analogy for understanding the difference between llms-full.txt and llms.txt is comparing it to a book and your study notes.

llms.txt is a concise version that highlights the most essential parts of your site. Even though there’s no strict rule on how long it should be, keeping it around 5-10 links is suggested. It’s like getting a quick overview of everything your site offers in just one file!

On the flip side, llms-full.txt is the whole shebang—it’s like trying to pack every tiny detail of your website into one text file. That level of granular detail can be super helpful for website owners in influencing how LLMs perceive their website.

How Large Language Models Actually Use llms.txt

When a Large Language Model such as OpenAI 4o or Gemini encounters your website, here’s what happens:

The LLM first checks for a large language model text file
If found, it uses this as a guide to understand:
- Your site’s primary topics and themes
- Which content is most authoritative
- How different pages relate to each other
The AI then uses this understanding when:
- Answering questions that might reference your content
- Generating recommendations or summaries
- Deciding whether to include your pages in its knowledge base

Key Differences Between llms.txt and Traditional SEO Files

Feature	llms.txt	robots.txt	sitemap.xml
Primary Purpose	Guides AI understanding	Blocks or allows crawlers	Lists crawlable pages
Format	Markdown	Plain text	XML
Audience	LLMs (ChatGPT, Claude)	Search engine bots	Search engines such as Google or Bing
Content	Priorities + context	Access restrictions	URL inventory
Impact	AI recommendations	Search visibility	Indexing

In short:

robots.txt tells search engines what pages to crawl.
sitemap.xml provides a full index for search engine bots.
LLMs.txt is specifically designed for AI consumption, so models like ChatGPT or Claude can quickly summarise or generate content based on your site.

Creating Your llms.txt File: 5 Easy Steps

Step 1: Planning Your llms.txt Structure

✅ Before writing your file, conduct a content audit to identify:

🔄 Priority Content: Your 5-10 most valuable pages/resources
🔄 Secondary Content: Supporting material that adds depth
🔄 Content to Exclude: Outdated or low-quality pages
🔄 Content Relationships: How pieces connect (tutorial series, etc.)

Step 2: Writing the File in Proper Markdown Format

✅ Your large language model text should follow this exact structure



# [Your Website Name]  
  
> A concise, one-sentence description of your site's purpose and value proposition.  
  
[Optional: 2-3 sentence elaboration on your site's focus area and audience.]  
  
## Core Content Sections  
  
### [Section Name 1]  
- [Page Title](URL): Brief description of content and its value  
- [Page Title](URL): Why this matters to readers  
  
### [Section Name 2]   
- [Page Title](URL): Content summary and relevance  
- [Page Title](URL): Key takeaways or unique aspects  
  
## Content to Handle With Care  
- [Page Title](URL): Reason for caution (e.g., outdated, controversial)  
- [Directory Path]: Explanation of excluded content  
  
## Additional Context  
[Any other information that would help LLMs understand your content's relationships,
updates, or special considerations.]

Step 3: Refer to our llms.txt Sample Template

Here’s how a cooking blog might structure theirs:



# Plant-Based Kitchen  
  
> Authoritative resource for vegan recipes and cooking techniques since 2015.  
  
Our tested recipes and in-depth guides help home cooks master plant-based cooking, from quick weeknight meals to advanced techniques.  
  
## Recipe Collections  
  
### Beginner-Friendly Meals  
- [15 Minute Vegan Pasta](https://example.com/pasta): Perfect quick dinner with pantry staples  
- [Foolproof Tofu Scramble](https://example.com/tofu): Breakfast staple with customization guide  
  
### Advanced Techniques    
- [Artisan Vegan Cheeses](https://example.com/cheese): 30-day aging process explained  
- [Molecular Gastronomy](https://example.com/molecular): Plant-based versions of classic techniques  
  
## Articles to Update  
- [2018 Nutrition Guide](https://example.com/nutrition): Needs new studies incorporated  
- /old-recipes/: Archive of untested user submissions  
  
## Content Relationships  
- Our "Vegan Basics" series ([Part 1](https://example.com/basics1), [Part 2](https://example.com/basics2)) should be read in sequence  
- Seasonal content is marked with [Summer], [Winter] etc. in titles

Step 4: Technical Implementation

✅ Uploading Your File:

🔄 File Creation:
- Use a plain text editor (VS Code, Notepad++)
- Save as llms.txt with UTF-8 encoding
🔄 Upload Methods:
- FTP: Upload to root directory (same level as robots.txt)
- cPanel: Use File Manager
- WordPress: Via SFTP or plugins like “File Manager”
- Static Sites: Commit to repository root
🔄 Verification:
- Visit yourdomain.com/llms.txt
- Check for proper Markdown rendering
- Validate with tools like MarkdownLint

Step 5: Keep Your llms.txt File Updated

Your website will change over time, so you need to update your llms.txt file regularly to keep your content accurate for chatbots.

✅ When to update?

When you add new product pages, blog posts, FAQs or documentation.
When you change your website structure.
When your chatbot is providing outdated information.

🔄 How to update?
Follow steps 1 to 4 again and replace the old llms.txt file with the new one. Don’t change the name of the new file, it should always be called llms.txt with no prefix or suffix

Advanced llms.txt Strategies

1. Content Prioritization Techniques

Use these methods to signal importance:

Order Matters: List the most important content first
Section Hierarchy: Use H2 for primary categories, H3 for subcategories
Strategic Linking: Include both deep links and category pages
Temporal Signals: Add “(Updated 2025)” for fresh content

2. Context Enhancement Methods

Help LLMs understand your content better by:

Defining Relationships: “This advanced guide builds on [Beginner’s Tutorial]”
Explaining Gaps: “Our 2021 study conclusions were updated in [2024 Research]”
Clarifying Intent: “Opinion piece representing author’s view”
Linking Related: “Companion video: [YouTube Tutorial]”

3. llms-full.txt for Comprehensive Documentation

For technical sites or documentation, consider adding an llms-full.txt that:

Includes complete API documentation
Provides detailed technical specifications
Offers comprehensive FAQ sections
Contains full glossary terms

Example Use Case:
A software company might use:

llms.txt for general navigation
llms-full.txt for complete API reference

Measuring the Impact of llms.txt

Key Performance Indicators

Track these metrics to assess effectiveness:

1. AI Referral Traffic

Monitor server logs for LLM user agents
Set up analytics segments for AI traffic

2. Content Visibility in AI Outputs

Test queries in ChatGPT/Claude
Track how often your content is cited

3. Engagement Metrics

Time on page from AI referrals
Conversion rates of AI-driven visitors

4. Search Performance

Changes in branded search volume
Featured snippet acquisition

Tools for Tracking

Google Analytics 4: Custom segments for AI traffic
Log File Analysis: Identify LLM crawlers
SEO Platforms: Monitor visibility changes
Custom Scripts: Track content citations in AI outputs

Real-World Examples of Websites That Use llms.txt

Cohere: Natural Language Processing (NLP) APIs
Cohere provides state-of-the-art NLP tools and APIs that allow developers to integrate text generation, classification, and semantic search capabilities into their applications. It is widely used for building AI-driven solutions like chatbots and content generation systems.

✅ Cohere llms.txt

2. HubSpot Developers: CRM and Marketing Automation
HubSpot uses LLMs to enhance customer relationship management and marketing automation. Their platform includes AI-powered chatbots, email personalization, and content recommendations to improve customer engagement and streamline marketing efforts..

✅ Hubspot Developers llms.txt
✅Hubspot Developers llms-full.txt

3. Hugging Face: Open-Source AI and Machine Learning Models
Hugging Face is a leader in the AI community, offering pre-trained LLMs and tools for developers to build and fine-tune models for tasks like text generation, translation, and sentiment analysis. It is a hub for open-source AI innovation.

✅ Hugging Face llms.txt

4. Zapier: Workflow Automation
Zapier integrates LLMs to automate workflows and connect apps. For example, LLMs can be used to summarize emails, generate responses, or extract key information from text as part of automated processes, making workflows more efficient.

✅ Zapier llms.txt
✅ Zapier llms-full.txt

5. FlowX.AI: Enterprise Workflow Automation
FlowX plays a critical role in AI and is used to optimize enterprise workflows by automating labor-intensive tasks, facilitating intelligent document processing, and improving communications between customers and businesses through large-organizational AI solutions.

✅ FlowX AI llms.txt
✅ FlowX AI llms-full.txt

6. ClearTax: Tax Filing and Financial Services
ClearTax uses AI and LLMs to make tax filing, financial planning, and tax compliance easy for individuals and businesses. Their solution automates everything from complex calculations to personalized recommendations and accurate submissions of taxes.

✅ Cleartax llms.txt

7. Cloudflare: Web Security and Performance
LLMs used by Cloudflare to improve web security and performance. Their tools powered by AI, assist in detecting and mitigating cyber threats, optimizing website performance, and providing intelligent insights for managing web traffic.

✅ Cloudflare llms.txt
✅ Cloudflare llms-full.txt

Future-Proofing Your llms.txt Strategy

Emerging Best Practices

1. Multimodal Content Signaling

Adding references to videos, infographics
Describing visual content for AI understanding

2. Temporal Context

Clear dating of content
Version history for evolving topics

3. Knowledge Graph Integration

Aligning with schema.org markup
Connecting to external authority sources

4. Personalization Hints

Audience segmentation cues
Prerequisite knowledge indicators

The Road Ahead

As AI search evolves, expect:

More sophisticated file formats
Two-way communication protocols
Standardized evaluation metrics
Automated optimization tools

Conclusion: Your AI Visibility Blueprint

Implementing llms.txt represents one of the most impactful yet overlooked opportunities in modern web strategy. By taking the time to:

Properly structure your file
Thoughtfully prioritize content
Provide meaningful context
Maintain and update regularly

You position your website to thrive in an increasingly AI-dominated information landscape. The simple act of adding this file can mean the difference between being invisible to LLMs and becoming a go-to resource for AI-generated answers.

Next Steps:

Conduct a content audit to identify priority material
Draft your llms.txt using our template
Upload and verify proper implementation
Monitor results and refine quarterly

Remember – in the age of AI search, visibility isn’t just about being found; it’s about being understood. Your large language model text file ensures AI systems comprehend and properly represent your content to the world.