Sitemap, Robots.txt & LLMs.txt Generator

What is a sitemap?

An XML sitemap is a file (usually at /sitemap.xml) that lists the pages on your site you want search engines to know about. Each entry can include when the page last changed, how often it changes, and a relative priority. A sitemap does not force anything to be indexed — it is a hint that helps crawlers discover pages, especially on large sites or sites with few internal links.

What is robots.txt?

robots.txt is a plain-text file at the root of your site (/robots.txt) that tells crawlers which paths they may or may not request. It is a convention that well-behaved crawlers follow; it is not a security control and does not hide a page from people who know the URL. It is also where you can point crawlers at your sitemap with a Sitemap: line.

What is llms.txt?

llms.txt is a proposed convention (llmstxt.org) for a Markdown file at /llms.txt that gives large language models a short, curated map of your site: a title, a one-line summary, and lists of the pages worth reading, each with a link and an optional description. It is a hint for AI assistants, not a crawl-control file — think of it as a hand-written table of contents aimed at machines. Adoption is still early, so treat it as a nice-to-have alongside your sitemap and robots.txt.

Sitemap vs robots.txt

They solve opposite problems. A sitemap says “here are pages worth looking at.” robots.txt says “please don’t crawl these paths.” Most sites want both: a sitemap listing real content, and a small robots.txt that keeps crawlers out of admin, login, and internal-search pages while pointing them at the sitemap. Blocking a page in robots.txt does not remove it from search results if other sites link to it — use anoindex meta tag for that.

How to use your generated files

1. Download sitemap.xml, robots.txt, and/or llms.txt from this page.
2. Upload each file to the root of your website, so it is reachable at https://yourdomain.com/sitemap.xml, https://yourdomain.com/robots.txt, or https://yourdomain.com/llms.txt.
3. Open each URL in a browser to confirm it loads and shows the expected content.
4. Optionally submit the sitemap URL in Google Search Console and Bing Webmaster Tools.

Common mistakes

• Accidentally blocking the whole site. A stray Disallow: / underUser-agent: * removes your site from search results. This tool warns you when that happens.
• Invalid or relative sitemap URLs. Sitemap entries must be full, absolutehttp(s) URLs — /about is not valid.
• Forgetting to publish the files. Generating them is not enough; they have to be uploaded to your site’s root.
• Wrong sitemap location. The Sitemap: line in robots.txt must be the real, public URL of the sitemap file.
• Duplicate URLs. Listing the same page many times adds noise. This tool removes exact and trailing-slash duplicates and tells you how many it dropped.