Robots.txt

Robots.txt - SEO Glossary

The robot.txt represents computer data. More specifically, it is a text file that is used at the website level to indicate to crawlers (mainly those of search engines) how they should explore the pages of websites.

More technically, they are part of an exclusion protocol and represent the standards that govern how crawlers browse the web, and access and index content.

It is included in the root of your website so that if the address of your site is: www.mysite.com, the URL leading to the robots.txt is:

  • www.mysite.com/robots.txt