Skip to content
Website glossary· Showing up on Google and in AI answers

What is robots.txt?

robots.txt is a small text file at the root of a website that tells search engine crawlers which parts of the site they may and may not crawl.

Why it matters for a small business site

One wrong line can hide a whole site from Google. A leftover “Disallow: /” from while the site was being built is a classic. Used well, it keeps crawlers out of places that don’t need them, like admin areas and internal search results.

How to check yours

  1. Open yourdomain.com/robots.txt in a browser.
  2. Look for “Disallow: /” under “User-agent: *”. That line blocks everything.
  3. In Google Search Console, the robots.txt report shows the version Google fetched and any problems with it.

What good looks like

Short and deliberate: everything public allowed, only private or useless areas blocked, and a Sitemap: line pointing to your sitemap.

Common mistakes

  • Blocking the whole site after launch.
  • Using it to keep a page out of Google. It stops crawling, not indexing, so a blocked page can still be listed if other sites link to it. Use a noindex tag instead, and don’t block that page, or Google never sees the tag.
  • Blocking the CSS and JavaScript files your pages need to display.
  • Treating it as security. The file is public, and it lists the paths you wanted hidden.
Put it into practice

Try it on a design of your own

SketchXFlow’s Google and AI search panel writes a robots.txt that lets search engines in and points them to your sitemap.

Try:
Not a page?LogoEmailVideo
Designs shared by the communitySee what people made — with the prompt behind each one →Featured onThe Design Tools