seo Question posted in
Some basic information to get you started with this can be found here.

How to update/replace robots.txt file in aws cloudfront – SEO

I have a website www.example.com. So when I access robots.txt file like www.example.com/robots.txt and it shows multiple lines text which is created/prepared by SEO team.

So there is a sub domain assets.example.com which is pointing to CloudFront. When I access the robots.txt file through CloudFront URL like https://assets.example.com/robtos.txt then it shows below result in the browser.

User-agent: *
Disallow: /

So there is a request to update the robots.txt file content in AWS CloudFront so https://assets.example.com/robtos.txt & https://www.example.com/robtos.txt should show the same text. I didn’t find anywhere robots.txt is placed in cloud-front.

Is it possible to update robots.txt in cloudfront? Is there any role of CloudFront here? Or we need to update the robots.txt for assets.example.com same as configured for example.com?

Please help me out. I’m very confused here.

Tags:

2

Answers


  1. 0

    You can point the CloudFront distribution (assets.example.com), then add a new origin with the domain name www.example.com then add a new cache behavior with the path pattern robots.txt and add the origin to it.

    This setup takes a request to assets.example.com/robots.txt and forwards it to www.example.com/robots.txt. With this, you can remove the duplications.

    Login or Signup to reply.
  2. 0

    You can invalidate cached by option invalidation.

    You can do following :

    1. Directly deploy build folder to S3 bucket.
    2. Not required to cached robots.txt file
    3. Whenever you deployed or upload build to S3,do the following step
    4. Go to cloudfront
    5. Do invalidation of objects
    6. Create entry /*
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search