What are Robots.txt files?
Robot exclusion standard or simply robots.txt is a standard used by websites to communicate with web crawlers and other web robots.In a robots.txt file with multiple user-agent directives, each disallow or allow rule only applies to the useragent(s) specified in that particular line break-separated set. If the file contains a rule that applies to more than one user-agent, a crawler will only pay attention to (and follow the directives in) the most specific group of instructions.
Top 10 Search engine and there Robots and Bots name
1. Googlebot by Google –
Googlebot is Google’s web crawling bot (sometimes also called a “spider”). Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site. As Googlebot visits each of these websites it detects links (SRC and HREF) on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.
2. Baiduspider by Baidu–
Baiduspider is a robot of Baidu Chinese search engine. Baidu (Chinese: 百度; pinyin: Bǎidù) is the leading Chinese search engine for websites, audio files, and images.
3. Bingbot by Bing-
Retired October 2010 and rebranded as Bingbot, this is a web-crawling robot (type of Internet bot), deployed by Microsoft to supply Bing (search engine). It collects documents from the web to build a searchable index for the Bing (search engine).
4. Yandex Bot by Yandex –
Yandex bot is Yandex’s search engine’s crawler. Yandex is a Russian Internet company which operates the largest search engine in Russia with about 60% market share in that country. Yandex ranked as the fifth largest search engine worldwide with more than 150 million searches per day as of April 2012 and more than 25.5 million visitors.
5. Soso Spider by Tencent used for search engine soso.com –
Soso.com is a Chinese search engine owned by Tencent Holdings Limited, which is well known for its other creation QQ. As of 13 May 2012, Soso.com is ranked as the 36th most visited website in the world and the 13th most visited website in China, according to Alexa Internet. On an average, Soso.com gets 21,064,490 page views everyday.
6. Exabot by Exalead –
Exabot is the crawler for ExaLead out of France. Founded in 2000 by search engine pioneers, Dassault Systèmes, ExaLead provides search and unified information access software.
7. Sogou Spider by a chines browser names sogou.com –
Sogou.com is a Chinese search engine. It was launched August 4, 2004. As of April 2010, it has a rank of 121 in Alexa’s Internet rankings. Sogou provides an index of up to 10 billion web pages.
8. Slurpbot by YAHOO -
Slurpbot is a bot which crawle the search results for Yahoo search engine.
Slurp does the following:
- Collects content from partner sites for inclusion within sites like Yahoo News, Yahoo Finance and Yahoo Sports.
- Accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users.
9. Facebook External Hit by facebook-
Facebook allows its users to send links to interesting web content to other Facebook users. Part of how this works on the Facebook system involves the temporary display of certain images or details related to the web content, such as the title of the webpage or the embed tag of a video. The Facebook system retrieves this information only after a user provides a link.
10. DuckDuckBot by DuckDuckGo-
DuckDuckBot is the Web crawler for DuckDuckGo, a search engine that has become quite popular lately as it is known for privacy and not tracking you. It now handles over 12 million queries per day. DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (their crawler) and crowd-sourced sites (Wikipedia). They also have more traditional links in the search results, which they source from Yahoo!, Yandex and Bing.
0 Comments