seo Question posted in
Some basic information to get you started with this can be found here.

Using noodp meta tag in a robots.txt file – SEO

Is it possible to add SEO tags such as ‘noodp’ to a robots.txt file instead of using <meta> tags? I am trying to avoid messing with our CMS template, although I suspect that I may have to…

Could I try something similar to this…

User-Agent: *
Disallow: /hidden
Sitemap: www.example.com
noodp:

I think robots.txt takes precedence over meta tags? For noindex for instance, the crawler will not even see the page in question. For something like noodp however, is this still the case?

Tags:

2

Answers


  1. 0

    No.

    Robots.txt file goal is to provide information to robots about crawl, not about what they should do on what they crawled.

    <meta> robots (or X-Robots-Tag instructions) and robots.txt instructions are two very distinct things.

    Google gives good information about this in his article Learn about robots.txt file:

    robots.txt should only be used to control crawling traffic

    If you want to add some robots instructions without messing with your CMS, HTTP header X-Robots-Tag might be a good solution. You can try to add it through your server config.

    Login or Signup to reply.
  2. 0

    You can’t do this with robots.txt, but you can get the same effect using the X-Robots-Tag response header.

    Add something like this to the appropriate part of your .htaccess file:

    Header set X-Robots-Tag "noodp"

    This tells the server to include the following line in the response headers:

    X-Robots-Tag: noodp

    Search engines (that support X-Robots-Tag) will interpret this header line exactly the same way they would interpret ‘noodp’ in a robots meta tag. In general, you can put anything in an X-Robots-Tag header that you can put in a robots meta tag. Note that the page must not be blocked by robots.txt, otherwise the crawler will never request the page, and will therefore never see the header.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search