Robots.text - The Robots.txt file is used to give instructions to search engine spiders about what they should and shouldn’t index when crawling the your website.