Generate Robots.txt for free according to your desired search engines. Many times Search engines like google, yahoo, etc are not able to read our site content due to faulty Robots.txt or blocker in it.
The robots.txt file, also known as the robots exclusion protocol or standard text file, this is a text file that tells search engine robots that which pages on your site are to be crawled and which not. Many websites don't need a robots.txt file. Because Google can usually find and is able to index all of the important pages on any site. And they'll also automatically NOT index the pages that aren't important or found duplicate versions of other pages.