Robots.txt test
View a website's robots.txt and test whether a path is allowed for Googlebot, Bingbot, or GPTBot.
About this tool
robots.txt is a text file in the root of a website that tells search robots which parts they may read. The file applies per domain, so exempel.se and www.exempel.se can have separate files.
The tool fetches the file and displays its contents. You can then enter a path and select a robot to see whether the path is allowed or blocked, and which rule applies. Rules are interpreted the same way Google does: the longest matching rule wins, and when rules are equal in length, Allow takes precedence.
Bear in mind that robots.txt only controls what robots can read, not what appears in search results. If you want a page excluded from Google, use noindex instead — and Google must be allowed to read the page for that to work.
Frequently asked questions
Do I need a robots.txt?
No. If the file is missing, all robots are free to read everything. You only need one if you want to block something or point to your sitemap.
How do I block AI robots like GPTBot?
Add a group with User-agent: GPTBot and Disallow: /. Robots that follow the rules will stop crawling the site. There is no enforcement, so not all robots will respect the file.
Why does a blocked page still appear in Google?
Google can show the URL if other sites link to it, even without having read the page. Remove the block and add noindex to the page instead.