The robot.txt represents computer data. More specifically, it is a text file that is used at the website level to indicate to crawlers (mainly those of search engines) how they should explore the pages of websites.
More technically, they are part of an exclusion protocol and represent the standards that govern how crawlers browse the web, and access and index content.
It is included in the root of your website so that if the address of your site is: www.mysite.com, the URL leading to the robots.txt is:
- www.mysite.com/robots.txt
Serving Your Location
Long Beach
Anaheim
Danville
Fremont
Fresno
Huntington Beach
Irvine
Los Angeles
Roseville
Orange County
Sacramento
San Diego
San Francisco
San Jose
San Mateo
Stockton
