Robots.txt is a text file that you can add to root for its website , that is, at the root of the server ( www.example.com/robots.txt ), directly below the domain .
Robots.txt instructs the search engines
The reason you often want a robots.txt is that indexing spiders on the web automatically search for one and thus give them the opportunity to instruct them. Common things that you use with your robots.txt file are, for example, asking a spider not to index the contents of some folders or to say that you have an XML sitemap , what it's called and where it's located.
Content in robots.txt
In its robot txt file, you write different things to allow or prohibit indexing. For example, a content may look like this:
For a full review, we would like to recommend the site www.robotstxt.org where all that is known is mentioned and explained. Worth knowing is that there are indexing c botts on the web that do not sound the instructions they try to give them in robots.txt but all serious web spiders do it.