seo Question posted in
Some basic information to get you started with this can be found here.

Prevent search engines from indexing my api – SEO

I have my api at api.website.com which requires no authentication.

I am looking for a way to disallow google from indexing my api.

Is there a way to do so?

I already have the disallow in my robots at api.website.com/robots.txt
but that just prevents google from crawling it.

User-agent: *
Disallow: /

The usual way would be to remove the Disallow and add a noindex meta tag but it’s an API hence no meta tags or anything.

Is there any other way to do that?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    It seems like there is a way to add a noindex on api calls.

    See here https://webmasters.stackexchange.com/questions/24569/why-do-google-search-results-include-pages-disallowed-in-robots-txt/24571#24571

    The solution recommended on both of those pages is to add a noindex meta tag to the pages you don't want indexed. (The X-Robots-Tag HTTP header should also work for non-HTML pages. I'm not sure if it works on redirects, though.) Paradoxically, this means that you have to allow Googlebot to crawl those pages (either by removing them from robots.txt entirely, or by adding a separate, more permissive set of rules for Googlebot), since otherwise it can't see the meta tag in the first place.


  2. 0

    It is strange Google is ignoring your /robots.txt file. Try dropping an index.html file in the root web directory and adding the following between the <head>...</head> tags of the web page.

    <meta name="robots" content="noindex, nofollow">
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search