Disallow is a directive that tells search engine robots not to crawl a page or website. In other words, you can use it to make Googlebot not go to certain areas of your website.
The Disallow directive is configured in the robots.txt file with the following syntax:
Disallow: [path], with the [path] as the beginning of the URL of the page not to crawl. Taking the example of definitions-seo, we have:
“ For a site like https://www.example.com/ :
Disallow:
No restrictions. Free access.
Disallow:/
The site is entirely prohibited for exploration.
Disallow: blog
No page whose URL begins with “blog” will be crawled (example: https://www.example.com/blog or https://www.example.com/blog/example.php).
Disallow: /*.pdf
No document whose URL contains “.pdf” will be explored (example: https://www.example.com/contract.pdf or https://www.example.com/blog/document.pdf or https://www.example.com/blog/document.pdf?langue). »
However, note that this directive does not prevent web pages from being indexed, which causes ” No information is available for this page ” messages to appear in search results.
References –
- https://moz.com/learn/seo/robotstxt
- https://developers.google.com/search/docs/advanced/robots/robots_txt
- https://searchfacts.com/robots-txt-allow-disallow-all
Other SEO Glossary
Allow | Disallow | NoIndex | Deindexing
