Meta robots is a meta tag that tells the search engines how to crawl and index the webpage and its content. It should be among the other meta tags in the head element of all web pages.
The meta robots tag can contain different parameters to define for the search engines how to crawl and index the contents of a web page. If parameters are detected by crawlers on the webpage, they give a strong suggestion of how they should crawl and index the content but that's not something that has to be followed. Some crawlers will therefore ignore your meta robots tags.
Here are the most common parameters that search engines use when crawling and indexing web pages:
Noindex: Tells the search engine not to index the web page.
Index: Tells the search engine to index the web page. This is the default mode and you do not need to specify this in the code.
Follow: Whether the webpage is indexed or not, the crawler will follow all links on the page, thus forwarding linkage to the linked pages.
Nofollow: Says to the search engine to not follow any links on the web page.
Noimageindex: Asks a crawler not to index any pictures on the web page.
None: Same meaning as putting noindex and nofollow together.
Noarchive: Search engines should not display a cached link to this page in the search results.
Nocache: Same as noarchive but used only by IE and MSN / Live.
Nosnippet: Search engines should not show a snippet (ie a meta description) for this page in the search result.
Unavailable_after: The search engine should not display the web page in search results after a certain date. The date must be written in RFC 850 format.
Meta robots out an SEO perspective
Search engine optimization is about making web pages available and as relevant as possible for crawlers. Therefore, it is important to ensure that all pages on the site are indexed and that links can be followed.
The parameters "index, follow" are the basic setting but the most common indexing error is to evaluate the robot tag instead of the parameter "noindex, nofollow". Usually this is a mistake from the web developer.
When you build and develop a website, you do not want it to be indexed in search engines before it's done. When you then launch the site, it's easy to forget to change the meta-robots tag.
You can check if your site is indexed by Google through a so-called site search: just type in site: https: //www.yourdomain.com in the search box, then the site's indexed pages will appear in search results.
However, all pages on your site need not always be indexed. For example, the shopping cart in an online store, thank you pages for filled out forms, etc. Be pages that you do not want to include in search engine indexes. These can then be set as noindex. The meta robot tag can also be a way to handle duplicate content .