Amazon Web Services Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – Proper downloading of files from S3 using pre-signed URLs

I am trying to refactor my current existing code base to download multiple files from S3. If I understand correctly the documentation, I am doing too many requests in a short period of time (1), (2) & (3).

Looking at the AWS SDK documentation it appears that a proper implementation exists, as described at:

However I have been unable to use it so far. I did check the example page:

See "Upload or download large files" at:

It looks like AmazonS3Client does not accept pre-signed URL as input (output only). Same story for TransferUtility

Anyone knows how to use AWS SDK (or c# alternative) to properly download multiple files from S3 using pre-signed URLs ? I have a reproducable case where a basic HTTP client setup fails for number of files equals 7000.

References:

  1. https://repost.aws/knowledge-center/http-5xx-errors-s3
  2. https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
  3. https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/retries-timeouts.html
Tags:

2

Answers


  1. 0

    you can clone your s3 bucket on your local directory which will automatically download your videos

    Login or Signup to reply.
  2. 0

    Yes, while working with a larger number of files such as 7000 using pre-signed URLs with AWS SDK is not the most suitable approach.

    You can use HttpClient with Exponential Backoff. But before that, you will be required to list all the pre-signed URLs for each file you want to download.

    HttpClient will help you to send individual download requests with each pre-signed URL and the Exponential backoff retry mechanism will help retry failed downloads.

    public async Task DownloadFilesWithBackoff(List<string> preSignedUrls)
{
    int retryCount = 0;
    TimeSpan delay = TimeSpan.FromSeconds(1);

    foreach (string url in preSignedUrls)
    {
        using (var httpClient = new HttpClient())
        {
            HttpResponseMessage response;
            while (true)
            {
                try
                {
                    response = await httpClient.GetAsync(url);
                    response.EnsureSuccessStatusCode();
                    break; // Success, exit loop
                }
                catch (HttpRequestException ex)
                {
                    retryCount++;
                    Console.WriteLine($"Download failed (attempt {retryCount}): {ex.Message}");
                    if (retryCount >= 3) // Adjust retry limit as needed
                    {
                        throw; // Re-throw exception after exceeding retries
                    }
                    await Task.Delay(delay);
                    delay *= 2; // Increase delay for next attempt
                }
            }
            // Process downloaded content from the response
        }
    }
}

    You can also consider implementing Multithreading with this which can help you download files concurrently.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search