Browse Article

Home » Articles tagged ‘disallow

How to Optimize “Robots.txt”

May 27, 2016 | Author: | Posted in SEO

CRB Tech Reviews would like to guide you about Robots.txt through this blog. As the name suggests, it is nothing but a text file which webmasters often create, to command search engine robots and crawlers like Google bot on ways to crawl & index pages on their website. To show signs of comprehending it, consider …