What is a robots.txt file?
Robots.txt is a text file webmasters create to instruct web robots (search engine robots) how to crawl pages on their website and which search engine can crawl there website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.
It also tells web robots which pages not to crawl.
Let’s say a search engine is about to visit a site. Before it visits the target page, it will check the robots.txt for instructions.
There are different types of Robots.txt for different activities on a site
1. To allow and disallow all search engine to index all the pages of your website:
- for allowing all search engine:
User-agent: *
allow: /
- for disallowing all search engine:
User agent: *
disallow: /
2. To allow and disallow a particular search engine to index all the pages of your website:
- for allowing a particular search engine:
User agent: search engine bot name
allow: /
- for disallowing a particular search engine:
User agent: search engine bot name
disallow: /
3. To allow and disallow a particular page to index on a search engine:
- To allow and disallow all the pages in all search engine:
User agent: *
allow: /
disallow: /
- To allow and disallow a particular page on all search engine:
User agent: *
allow: /page name you want to index/
disallow: /page name you doesn't want to index or show on search engine/
where you can see your Robot.txt files?
one can see there Robot.txt filles in" www.domainname.com/robots.txt", where domain name is the name and Url of a particular website whom robots file you want to see.
0 Comments