Hello friends,
today I am going to write regarding the thanks to add a robots.txt and
sitemap.xml into internet web site
First of all it’s
important to know what is Robots.txt and Sitemap.xml file. I justify a fast
introduction here for extra information regarding those searches Google.
Robots.txt
Each search engine have
its own crawlers like robots, scooters etc. When these crawlers go to any
internet web site first of all they check the robots.txt file and then crawl
the entire internet web site. In line with robots.txt file permission they
crawl an online web site. as an example if I prohibit the Google crawler to
contact us page then my contact us page is not crawled. This could be an idea
regarding Robots and why we have a tendency to tend to use the Robots.txt file.
Here is that the subsequent code that is used for robots.txt file.
user-agent:*
Disallow:
Use * and leave blank
field in front of disallow if you want to allow all robots to crawl your
internet web...
.jpg)
