skip to Main Content

Do you know if it is possible to force the robots crawl on www.domaine.com and not domaine.com ? In my case, I have a web app that has enabled cached urls with prerender.io (to view the HTML code), but only on www.

So, when the robots crawl on domaine.com, it has no data.

The redirection is automatic (domaine.com> http://www.domaine.com) on Nginx, but no results.

I said that my on my sitemap, urls have all www.

My Nginx redirect :

server {
  listen                *:80;

  server_name           stephane-richin.fr;

  location / {

    if ($http_host ~ "^([^.]+).([^.]+)$"){
      rewrite ^/(.*) http://www.stephane-richin.fr/$1 redirect;
    }

  }
}

Do you have an idea ?

Thank you !

2

Answers


  1. Could you have a robots.txt file with

    User-agent: *
    Disallow: /
    

    on domaine.com and a different one with

    User-agent: *
    Disallow:
    

    on http://www.domaine.com?

    Login or Signup to reply.
  2. If you submitted a sitemap with the correct URLs a week ago, it seems strange that the Google keeps requesting the old ones.

    Anyway – you’re sending the wrong status code in your non-www to www redirect. You are sending a 302 but should be sending a 301. Philippe explains the difference in this answer:

    Status 301 means that the resource (page) is moved permanently to a new location. The client/browser should not attempt to request the original location but use the new location from now on.

    Status 302 means that the resource is temporarily located somewhere else, and the client/browser should continue requesting the original url.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search