Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website (yoursite.com/robots.txt) that tells search engine crawlers which pages they are allowed or not allowed to visit. It's the first file Google, Bing, and AI crawlers check before crawling your website.
Does robots.txt affect Google rankings?
Yes, indirectly. If you accidentally block important pages in robots.txt, Google can't crawl or index them — meaning they won't appear in search results at all. It's one of the most common technical SEO mistakes. Use this generator to make sure you're only blocking what you intend to.
Should I block AI crawlers like GPTBot?
It depends on your goals. If you want your brand to appear in ChatGPT, Gemini, and Perplexity answers, you should allow AI crawlers. If you're concerned about your content being used for AI model training without attribution, you can block training-specific bots. Note that blocking crawlers doesn't remove you from AI answers if your content is already indexed.
What's the difference between Disallow and noindex?
robots.txt Disallow prevents crawlers from visiting a page entirely. The noindex meta tag allows crawlers to visit a page but tells them not to include it in search results. If you Disallow a page, Google can't see the noindex tag — so use Disallow for pages you don't want crawled at all, and noindex for pages you want crawled but not shown in search.
Where do I upload my robots.txt file?
Your robots.txt must be placed in the root directory of your website, accessible at https://yoursite.com/robots.txt. In WordPress, plugins like RankMath and Yoast SEO let you edit it from the dashboard. For other platforms, upload via FTP or your hosting file manager.
Can I have multiple User-agent rules?
Yes. You can create separate rules for different crawlers. For example, you can allow Googlebot everywhere but block GPTBot from your entire site. This generator handles this automatically when you enable the AI crawler blocking option.