What is robots.txt?

A robots.txt is a small text file that lists which pages and files a search engine may or may not crawl. In other words, you give instructions to search engine bots to indicate which pages of your website may appear in search results. Where should it be? The robots.txt file must be in the root of your domain. […]