Even if you don’t think you need it, it’s still good practice to provide a “robots.txt” file in the root directory of your site for search engine spiders to find. Not only will it remove the 404s from your error_log (happens every time a spider/bot looks for it and it doesn’t exist), but it also provides a quick and efficient way to block certain sections of your site from being indexed. This is by far a better method than adding rel=”nofollow” to your links or the following meta tag to the header of each page in question.
<meta name="robots" content="noindex, nofollow" />
The most basic robots.txt file would include the following. This tells the search engines to index everything it can find.