Question 259156 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – wait for a promise of promises

Im trying to download a bucket from S3

async function downloadBucketRaw(buckt, prefix) {
  return s3.listObjectsV2({ Bucket: buckt, Prefix: prefix },
    (err, obj) => {
        if (err) return console.log(err);

        var promises = []

        obj.Contents.forEach(element => {
            if (element.Size > 0) {
                var p = downloadFileFromS3({ Bucket: buckt, Key: element.Key });
                promises.push(p)
            }
        });
        return promises
    });
  }
}

async function downloadFileFromS3(item) {

    var file = s3.getObject(item,
      (err, data) => {
        if (err) throw err;
    });

    file.createReadStream()
      .pipe(fs.createWriteStream(item));

    return file.promise()
}

In order to wait for all the downloads to happen I want to do something like:

Promise.all(downloadBucketRaw(buckt, prefix))

But it is not working, I am not sure about how to do a wait of a promise returning a list of promises

Tags:

2

Answers


  1. 0

    The issue in your code is that you’re not properly handling promises in the downloadBucketRaw function. The s3.listObjectsV2 method does not return a promise, so you can’t directly use async/await with it.

    To fix this, you can wrap the callback-based function with a Promise and resolve it once the callback is called. Here’s an updated version of your code:

    You can rewrite your code like below

    function downloadBucketRaw(buckt, prefix) {
    return new Promise((resolve, reject) => {
s3.listObjectsV2({ Bucket: buckt, Prefix: prefix }, (err, obj) => {
  if (err) {
    reject(err);
  } else {
    var promises = [];

    obj.Contents.forEach(element => {
      if (element.Size > 0) {
        var p = downloadFileFromS3({ Bucket: buckt, Key: element.Key });
        promises.push(p);
      }
    });

    resolve(promises);
  }
});});}


   async function downloadFileFromS3(item) {
 return new Promise((resolve, reject) => {
s3.getObject(item, (err, data) => {
  if (err) {
    reject(err);
  } else {
    var file = data.Body;

    file.createReadStream().pipe(fs.createWriteStream(item));

    resolve(file.promise());
  }
});});}

downloadBucketRaw(buckt, prefix).then(promises => Promise.all(promises)).then(res => console.log(res)).catch(err => console.log(err));
    Login or Signup to reply.
  2. 0

    The s3.listObjectsV2() method returns an AWS.Request, which provides a .promise() method, which turns the Node callback style into a then-able Promise:

    async function downloadBucketRaw(buckt, prefix) {
  try {
    var obj = await s3.listObjectsV2({
      Bucket: buckt,
      Prefix: prefix
    }).promise(); // Convert into a Promise, which can be await-ed
  } catch (err) {
    console.log(err);
    return;
  }

  return Promise.all(
    obj.Contents
      .filter((element) => element.size > 0)
      .map((element) => downloadFileFromS3({
        Bucket: buckt,
        Key: element.Key
      }))
    );
  }
}

// Usage:
await downloadBucketRaw(buckt, prefix);

    Note that if you were to return the Array of Promises directly, you would have to use it like:

    await Promise.all(await downloadBucketRaw(buckt, prefix)); // Extra await, to get rid of the extra Promise
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search