The robots.txt file tells search engines which parts of your website they are allowed to visit. It lives in the root folder of your website and can be accessed at mittforetag.se/robots.txt.
A mistake in robots.txt can remove your entire website from Google. Save a copy of the old file before making any changes.
How to edit the file
- Log in to DirectAdmin via the customer portal. See How do I log in to my control panel?
- Go to System Info & Files and select File Manager.
- Open the folder for your domain name, then open the public_html folder.
- If robots.txt already exists, double-click it to open it. Otherwise, create a new file with that name.
- Make your changes and click Save File.
You can also upload the file using an FTP client. See Using an FTP client (FileZilla).
A good starting point for WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://mittforetag.se/sitemap_index.xmlReplace the Sitemap address with your own. See Submitting a sitemap to Google.
If there is no robots.txt in public_html, WordPress will generate a basic one automatically. If you upload your own file, that one takes precedence.
Common mistakes
- Disallow: / under User-agent: * blocks all search engines from the entire website. This line is sometimes used while a site is being built and then forgotten.
- Robots.txt does not remove pages that are already indexed in Google. For that, use the noindex tag on the page.
- The file does not protect secret pages. It is a set of instructions that reputable search engines follow, but anyone can read it.
After making changes, check that the file looks correct at mittforetag.se/robots.txt. In Google Search Console you can see whether Google has fetched the new version.
Robots.txt in other content management systems
Joomla
- Open File Manager in DirectAdmin and navigate to the public_html folder.
- Joomla comes with a file called robots.txt.dist. If no robots.txt exists, rename it to robots.txt.
- Open the file and add a line with your sitemap at the bottom, for example Sitemap: https://mittforetag.se/sitemap.xml.
- Save the file.
Drupal
Drupal includes a ready-made robots.txt in the website's root folder, usually public_html or public_html/web. Edit it in File Manager in the same way. Bear in mind that the file may be overwritten when Drupal is updated. If you make extensive changes, the RobotsTxt module saves the content to the database instead.
Wix, Squarespace, and Shopify
- Wix: robots.txt is edited under the SEO settings, in the robots.txt editor.
- Squarespace: the file is generated automatically and cannot be edited. Pages are hidden instead using the search engine visibility setting on each page.
- Shopify: the file is generated automatically. It can be customised by adding the robots.txt.liquid template to your theme, but this requires knowledge of Shopify theme development.
Wix, Squarespace, and Shopify run on their respective providers' own servers, not on your web hosting. Menu names may change over time, so search in their help centres if you cannot find the setting.
Menu names in Joomla and Drupal are usually the same in English and Swedish. If your control panel looks different, it may be due to the version you are running.