The Ultimate Guide to the Robots Meta Tag for Developers

The robots meta tag provides instructions to web crawlers (robots). It is an alternative to a robots.txt file and is implemented on a page-by-page basis by adding a meta tag to the head of an HTML document.

You won't find the robots meta tag mentioned in HTML5 because it isn't formally part of the specification. The tag was proposed during a W3C workshop in 1996 and it's use was explained in Appendix B of HTML 4.01 in December of 1999. However, the meta robots tag has never been officially added to the HTML specification.

That doesn't mean that search engines don't care about the tag. While unscrupulous web crawlers can (and do) ignore the tag, search engine web crawlers look at the robots meta tag for instructions on how to treat the contents of a webpage.

Basic Syntax

The basic syntax of the meta robots tag is quite simple:

<meta name="robots" content="instructions go here">

To use it, added the tag to the head element of every webpage that needs to provide instructions for web crawlers. Unlike the robots.txt file which provides sitewide instructions, the robots meta tag only applies to the page on which it appears.

Robots.txt vs Robots Meta Tag

On the surface, robots.txt seems strategically superior to the robots meta tag because robots.txt keeps the web crawler instructions for an entire site in one place.

Unfortunately, search engines don't care about convenience. When it comes to keeping URLs from being indexed, using disallow in robots.txt is not as effective as using noindex in a robots meta tag.

Don't believe us? Then believe SEO experts Moz. They've presented a compelling case for choosing the robots meta tag over robots.txt.

Telling Robots What to Do

In the syntax example we used "instructions go here" as a placeholder. There are several different values that can be used in their place.

Index and Noindex

One thing you can tell a web crawler is whether or not you want the page indexed.

<!-- index this page, the default behavior -->
<meta name="robots" content="index">

<!-- don't index this page -->
<meta name="robots" content="noindex">

By default, web crawlers assume they have a green light to index every webpage. So if you want a page indexed, just omit the robots meta tag. However, if you don't want a page indexed add the tag with a noindex value.

Follow and Nofollow

By default, web crawlers follow every link on a webpage and index those linked pages (unless they have instructions that prevent indexing). So adding a robots meta tag with the follow value is optional. However, use nofollow if you don't want web crawlers to follow the links on a page.

<!-- follow all links, the default behavior -->
<meta name="robots" content="follow">

<!-- don't follow any links -->
<meta name="robots" content="nofollow">

Instructions can piggyback on each other as well. So if you want to add both nofollow and noindex to a webpage, you can add both at once like this:

<meta name="robots" content="noindex, nofollow">

All or None-Thing

The following combinations are common:

  • index and follow
  • noindex and nofollow

You can use the shorthand all or none instructions to make this happen.

<!-- equivalent to content="follow, index", the default behavior-- >
<meta name="robots" content="all">

<!-- equivalent to content="nofollow, noindex"-- >
<meta name="robots" content="none">

Specialized Meta Robots Tags

Use name="robots" and you'll be addressing every web crawler that's listening. However, you can target one web crawler at a time.

<!-- instructions for google's web crawler -->
<meta name="googlebot" content="instructions go here">

<!-- instructions for yahoo's web crawler -->
<meta name="slurp" content="instructions go here">

<!-- instructions for bing and msn web crawlers -->
<meta name="bingbot" content="instructions go here">

<!-- instructions for ask's web crawler -->
<meta name="teoma" content="instructions go here">

There are additional keywords you can use to deliver targeted instructions. However, it's worth noting that not all search engines pay attention to all of these commands.

  • content="noimageindex": instructs web crawlers not to index any images that appear on a webpage. However, if those images appear on other webpages they will be indexed. To prevent indexing of images add robot instructions to the HTTP header delivered with the image file.
  • content="noarchive": instructs web crawlers to index the webpage without caching a complete copy of the webpage.
  • content="nosnippet": instructs search engines not to display a snippet when the page is shown in the search results and also prevents caching of the page.
  • content="noodp": instructs search engines not to use the page description from ODP as the snippet in search results.
  • content="noydir": instructs Yahoo not to use the page description from the Yahoo Directory as the snippet in search results.
  • content="notranslate": instructs Google not to offer to translate the web page.
  • content="unavailable_after: [RFC 850 date/time]": instructs Google that the page should not appear in search engine results after a specific date and time.


Most robots meta tags are pretty simple. However, if you're planning on providing complex instructions here are a few resources where you can use to learn more about these tags.

Adding Robots Meta Tags to Your Website

If you want to add robots meta tags to your site you can copy the tags presented in this article and paste them into your site's HTML. In addition, there are tools you can use to generate custom instructions and to automatically add tags to webpages generated by content management systems.

Please, Control Your Robots

If you want to control how web crawlers index your site, the robots meta tag is a good mechanism. It's easy to understand, easy to implement, and can have a powerful impact on how search engines index your website.

Use it well and the robots meta tag will ensure search engines treat your site just the way you want them too. Use it poorly and search engines may forget all about you.

Just make sure you use it with care.

Further Reading and Resources

We have more guides, tutorials, and infographics related to coding and website development:

HTML for Beginners — Ultimate Guide

If you really want to learn HTML, we've created a book-length article, HTML for Beginners — Ultimate Guide And it really is the ultimate guide; it will take you from the very beginning to mastery.