Plesk Question posted in
The official documentation for Plesk Obsidian can be found here.

Throttling S3 commands with aws cli – Plesk

I’m running a backup script using AWS CLI to perform an S3 sync command every night on my MediaTemple server. This has run without fail for months, but I updated my Plesk installation and now every night, when the backup script runs, MediaTemple disables my server due to excessive usage. The limits I seem to be crossing are as follows:

RESOURCE INFO:
Packets per second limit: 35000
Packets per second detected: 42229.11667000000306870788
Bytes per second limit: 50000000
Bytes per second detected: 61801446.10000000149011611938

They also include a networking snapshot at the time they take the server offline which includes many open connections to Amazon IP addresses (9 at time of the snapshot).

Is there anything I can do to throttle the connections to AWS? Preferably I’m looking for an option within the AWS API (though I haven’t seen anything useful in the documentation), but barring that, is there something I can do on my end to manage the connections at the network level?

Tags:

5

Answers


  1. Chosen as BEST ANSWER
    0

    I ended up using Trickle and capping download & upload speeds at 20,000 kb/s. This let me use my existing script without much modification (all I had to do was add the trickle call to the beginning of the command).

    Also, it looks like bandwidth throttling has been added as an issue to AWS CLI, so hopefully this will all be a non-issue for folks if that gets implemented.


  2. 0

    The AWS CLI S3 transfer commands (which includes sync) have the following relevant configuration options:

    • max_concurrent_requests –
      • Default: 10
      • The maximum number of concurrent requests.
    • multipart_threshold –
      • Default: 8MB
      • The size threshold the CLI uses for multipart
        transfers of individual files.
    • multipart_chunksize –
      • Default: 8MB
      • When using
        multipart transfers, this is the chunk size that the CLI uses for
        multipart transfers of individual files.

    This isn’t so granular as throttling packets per second, but it seems like setting a lower concurrent request value and lowering both multipart threshold and chunksize will help. If the values you pasted are close to average, I would start with these values and tweak until you’re reliably not exceeding the limits anymore: 

    $ aws configure set default.s3.max_concurrent_requests 8
$ aws configure set default.s3.multipart_threshold 6MB
$ aws configure set default.s3.multipart_chunksize 6MB
    Login or Signup to reply.
  3. 0

    If you can not make trickle work with aws s3 command like me, you may use:

    sudo apt-get install pv (or yum install pv)
    pv -L 1M local_filename 2>/dev/null | aws s3 cp - s3://bucket_name/remote_filename

    where -L 1M limits the bandwidth to 1M/s
    and the dash after cp indicate stdin

    Note: the awscli from apt-get is too old to support the stdin input, you need to upgrade it via pip

    Login or Signup to reply.
  4. 0

    As well as changing the max default connections and chunk size already mentions you can also set the max_bandwidth. This is very effective when uploading large single files.

    aws configure set default.s3.max_bandwidth 50MB/s

    Login or Signup to reply.
  5. 0

    I could not get trickle to work with aws-cli, but came across s3cmd which works great for me. It has an option to rate limit. It can be found in the Fedora repos, and I imagine other distros have it packaged too.

    s3cmd --progress --stats --limit-rate=50k sync ./my_photos/ s3://mybucket

    s3cmd man page

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search