Nginx Question posted in
The official documentation can be found here.

Nuxt robots.txt using nginx & pm2

I’m using nginx and pm2 to serve a nuxt app (SSR).
I tried to use the sitemap module and the robot.txt module.

The sitemap module works fine and is accessible at /sitemap.xml.

However the robots.txt file is not accessible through nginx. Using nuxt build and nuxt start in local it works fine and I have access to /robots.txt. But using nginx I have a 404. Not the Nuxt 404 but the nginx one. All other inexistant url show the Nuxt 404

Using Nuxt SSR and the robot module does not generate a robots.txt file. It’s a middleware and response to the /robots.txt url is dynamically generated on each request

Why do I have that 404 with the robot module but not with the sitemap?

Thanks for your help.

Tags:

2

Answers


  1. 0

    To quickly serve a robots.txt from Nginx without actually having access to the physical file you can define the content of the robots.txt file in the Nginx .conf file.

    Allow access to all User-agents:

    location /robots.txt {return 200 "User-agent: *nDisallow:n";}

    Disallow access to every User-agent:

    location /robots.txt {return 200 "User-agent: *nDisallow: /n";}
    Login or Signup to reply.
  2. 0

    I’m using Google Cloud App Engine (GAE) which also uses Nginx to host my Nuxt.js app and had the same problem as yours – I can access my sitemap.xml unlike robots.txt.

    For GAE configuration I use an app.yaml file where among all configurations you can set handlers for static files. You can explicitly set that static files requests will be serve from a Nuxt’s ‘static’ folder. As always types of potentially requested files are set as a regexp that shows a list of file extentions (e.g. .(gif|png|txt)$).

    In my case I have a ‘.txt’ file among other static files to serve from Nuxt’s ‘static’ folder. That’s why Nginx was looking for my robots.txt directly in the ‘static’ folder and did not give a chance for Nuxt to generate it.
    So when I removed the ‘.txt’ extention everything work fine and now I can get my robots.txt.

    I’m not familiar with Nginx configurating, but maybe you should check what extentions of static files it is serving and exclude a ‘.txt’ extention serving from Nuxt’s ‘static’ folder.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search