seo Question posted in
Some basic information to get you started with this can be found here.

Can Search Engine read robots.txt if it's read access is restricted? – SEO

I have added robots.txt file and added some lines to restrict some folders.Also i added restriction from all to access that robots.txt file using .htaccess file.Can Search engines read content of that file?

Tags:

3

Answers


  1. 0

    This file should be freely readable. Search engine are like visitors on your website. If a visitor can’t see this file, then the search engine will not be able to see it either.

    There’s absolutely no reason to try to hide this file.

    Login or Signup to reply.
  2. 0

    Web crawlers need to be able to HTTP GET your robots.txt, or they will be unable to parse the file and respect your configuration.

    Login or Signup to reply.
  3. 0

    The answer is no! But the simplest and safest too, is still to try:
    https://support.google.com/webmasters/answer/6062598?hl=en

    The robots.txt Tester tool shows you whether your robots.txt file
    blocks Google web crawlers from specific URLs on your site. For
    example, you can use this tool to test whether the Googlebot-Image
    crawler can crawl the URL of an image you wish to block from Google
    Image Search.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search