Why is Robots.txt so important?
Robots.txt is a file on your website that tells search engine crawlers the areas on your site they should avoid and the areas they can visit. With proper knowledge of the format, this file can be very useful to website owners, but it has its dangers if used incorrectly.
It is important to create the file in the root directory of the website so that the search engine can crawl it. Knowing the value, use, and SEO preferences of a robots.txt is important for any website.
What’s Robots.txt?
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. It is mainly used to avoid overloading a website with requests.
In practical use, the robots.txt file is used to let search engine crawlers allow or restrict indexing selected areas of the website.
How Robots.txt work?
The robots.txt file can be used to block specific content on a website from being listed in web search results. For example, if you have videos on your website that you don't want Google or other search engine to list, then the robots.txt file can be used to block them from being crawled so they won't show up.
If you have a website, you can check if your site has a Robots.txt file by adding /robots.txt to the end of your domain name; for example: https://www.example.com/robots.txt. Search engines read this file first, and based on the rules set, the crawlers will collect and index the pages.
Robots.txt accepted syntax
The standard syntax that is added in the robots.txt file comprises of the below entities.
User-agent: The specific web crawler that you are providing crawl instructions.
Disallow: The command tells a user-agent not to crawl a URL. “Disallow:” can only be used once per URL.
Allow: This command tells Googlebot it can access an URL or subfolder. This is true even if parent page or subfolder are disallowed.
Sitemap: This specifies the path to the Sitemap.xml file. You’ll find more information about sitemap.xml here.
Basic guidelines for creating a robots.txt file
Making a robots.txt file accessible and useful is pretty easy. Here's what you need to do:
Open a text editor.
Add rules to the file.
Save file and name it as robots.txt.
Upload the robots.txt file to the root of your site.
Test the robots.txt file.
How to add robots.txt rules
Rules are instructions for Crawlers about what parts of your site they can crawl. To make sure that your site isn't blocked and abused, follow these guidelines when adding a rules to your robots.txt file:
Robots files are groups of rules that can also include more than one group.
There are multiple rules (also known as directives), one rule per line. Each group begins with a User-agent line that specifies the target of the groups. This ranges from devices to types of files and more.
The group gave the following information:
To which user agent the group rules applies to.
The directories or files that they can access.
The directories or files that they cannot access.
A user agent can only match one rule set, and if two or more rules match it, the rules will be combined into a single group before processing.
One assumption is that a user agent should be allowed to run on any page or directory, not blocked by a disallow rule.
Rules are case-sensitive.
Use # character to the begin a comment. Comments are ignored by the crawlers.
Google's crawlers support the following commands in robots.txt files:
user-agent: Google user agent names are listed in the Google list of user agents. Using an asterisk (*) matches all crawlers except the various AdsBot crawlers, which must be named explicitly.
disallow: A directory or page, relative to the root domain, you do not want the user agent to crawl. It must start with a / character and if it refers to a directory, it must end with a / mark.
allow: A directory or page, relative to the root domain, that may be crawled by the user agents. This is used to override a disallow rule to allow crawling of a subdirectory or page in a disallowed directory.
sitemap: The location of a sitemap for this website. The sitemap URL must be a fully-qualified URL. Sitemaps are a good way to indicate which content Google should crawl, as opposed to which content it can or cannot crawl.
All rules, except sitemap, support the * wildcard for a path prefix, suffix, or entire string.
Lines that don't match any of these rules are ignored.
Upload the robots.txt file
Now the robots.txt file saved on your computer is ready to be uploaded to your website. Once it is uploaded, it will be made available to search engine crawlers. Connect to your website hosting and upload the robots.txt file in the root directory of your domain.
After you've uploaded the robots.txt file, test whether it's publicly accessible and if Google can parse it.
How to check if the robots.txt file is correct?
To verify and monitor your robots.txt file, you can use the testing tool for robots.txt provided by Google Webmaster Tools.
To use this tool, you first need to register your website. Once you do that, select your website from the list in Google Search Console and Google will return with possible troubleshooting steps and highlighted errors, if any.
Benefits of using robots.txt
Robots.txt file provides crawlers with instructions on what URLs they should index and display on their platform. This file provides the leverage to instruct to crawl specific URL or the entire website. Robots.txt has the link to the sitemap.xml acts as a index for search engines to find the location.
It is useful if you do not wish search engines to:
Allow / Restrict search engines to index specific URLs of the website or a whole website
Have custom rules based on the search engines.
Index / Block specific file formats (images, videos, PDFs)
Avoid duplicate pages from getting indexed.
Block staging / testing websites.
Block user profile links, and many more.
If you have a robots.txt file, then it'll be easy for search engines to find the XML sitemap for the latest content without having to crawl all your pages and run into them days later.