Question 19337 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

AWS S3: Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate

I setup AWS S3 acceleration in one of my buckets, mybioinformaticsm and use S3 acceleration to upload a big file from local to S3, as in a following command;

aws s3 cp /Volumes/Documents/R/BM/bm.int s3://myawsbioinfo/bm.int  --region us-east-2 --endpoint-url https://myawsbioinfo.s3-accelerate.amazonaws.com

It used to work until recently. When I use this command, it failed with a following error message;

Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate

I have not set up Dualstak and am not sure about FIPS. At this point, I would really appreciate any input on this.

Tags:

2

Answers


  1. 0

    I would suggest removing your bucket name from --endpoint-url and try just the following.

    aws s3 cp /Volumes/Documents/R/BM/bm.int s3://myawsbioinfo/bm.int --region us-east-2 --endpoint-url https://s3-accelerate.amazonaws.com

    AWS is likely (wrongly) determining your endpoint as a custom endpoint. It seems the direction of the various sdk’s is to dynamically determine endpoints based on profiles and options. Another option would be to turn accelerate on by default and do not include the --endpoint-url at all.

    See https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html

    Login or Signup to reply.
  2. 0

    To use S3 accelerate, I would suggest to not use the --endpoint-url argument. Instead, set the s3.use_accelerate_endpoint configuration to true. You can do this either with the command aws configure set default.s3.use_accelerate_endpoint true as documented here or by editing the config file directly. The CLI will then determine the correct endpoint to use. Based on the error message you report, you may already have the s3.use_accelerate_endpoint setting enabled.

    Some additional context and a method for debugging which setting is causing the error message:

    The AWS CLI v2 updated its logic for endpoint resolution in version 2.9.0 which was released on November 18, 2022. AWS CLI v1 received the same update earlier in version 1.26.0 on October 24, 2022. Endpoint resolution is the process by which the correct hostname for a given service and operation is determined based on inputs such as the region name and configuration settings. The new versions use endpoint ruleset files that are shared between AWS SDKs in different languages. For example, the S3 endpoint ruleset file is here. The error Host override cannot be combined with Dualstack, FIPS, or S3 Accelerate is defined in this file.

    For low level debugging purposes, it is possible to see what parameters the endpoint ruleset was invoked with by adding the --debug flag to the command:

    aws s3 cp /foo/bar s3://mybucketname/object.txt --region us-east-2 --endpoint-url https://mybucketname.s3-accelerate.amazonaws.com

    When run on a system with default configuration settings, the debug output contains these two lines:

    Calling endpoint provider with parameters: {'Bucket': 'mybucketname', 'Region': 'us-east-2', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://mybucketname.s3-accelerate.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
Endpoint provider result: https://mybucketname.s3-accelerate.amazonaws.com

    Note that 'UseFIPS': False, 'UseDualStack': False, and 'Accelerate': False and that an endpoint is returned and not an error. The reported error appears if one of these parameters is not False. The ruleset contains this error condition because, in the general case, it is impossible to know whether the value of --endpoint-url should take precedence over the S3 transfer acceleration setting, or vice versa. Of course, in this specific case the outcome would be the same.

    Enabling the S3 transfer acceleration setting and dropping the --endpoint-url argument results in these lines:

    Calling endpoint provider with parameters: {'Bucket': 'mybucketname', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': True, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
Endpoint provider result: https://mybucketname.s3-accelerate.amazonaws.com

    Note that this is the correct endpoint for using S3 transfer acceleration.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search