Robots.txt is a small piece of file which is used to control access to the website. By using robots.txt, you can easily prevent search engine bots to index certain directories.
It control the entire crawling of your site. But remenber spybot and other suspicious spider don’t follow the guidelines.
example-
User-agent: *
Disallow: /admin/
The above code will stop search engine to index ADMIN folder of your website.
Sitemap is created to tell the search bot about site structure, navigation, pages and more. They are generally created in XML format. You can also use .txt but .xml is used by almost all search engines.
So now, how to add its url in the robots.txt file?
User-agent: *
Disallow: /admin/
Sitemap: http://www.example.com/sitemap.xml
By this way, the search engine bot will automatically find the file location. Now there is no need to tell or add each search engine separately about sitemap. Uhhh….Easy?
Bing has asked webmasters to include a location of a sitemap in the file.
That’s all folks!! Keep visiting.
Twitter
Facebook
You-tube
Google
Blog
Hey nice article mate! Well with this help, I was able to add a sitemap to my robots file!