skip to Main Content

How can I write a stream to S3 and immediately retrieve a link to read the content back?

I’d like to upload my mp3 file to S3, get the URL to the resource, and send it to the user.

Since the upload may take upwards of 10 seconds, I’d like to stream the upload and allow the user to stream it from my bucket. However, the URL does not seem to be valid until the entire file is finished uploading.

  const passThroughStream = new stream.PassThrough();
  myAudioStream.pipe(passThroughStream);

  const uploadParams = {
    Bucket: bucketName,
    Key: key,
    Body: readable,
  };

  s3.upload(uploadParams, function (err, data) {
       console.log("full mp3 uploaded");
  });

  // Wait 500ms for first few bytes to upload before streaming the file
  await new Promise((resolve) => setTimeout(resolve, 500));
  const fileUrl = s3.getSignedUrl("getObject", {
    Bucket: bucketName,
    Key: key,
    Expires: 120,
  });
  console.log("File URL", fileUrl);

If I click into this URL, I see the following.

enter image description here

I cannot access the resource until the full 10 seconds are finished. I assumed I would be able to access it the moment I began the upload.

2

Answers


  1. That is correct. Objects in Amazon S3 are either ‘all there’ or ‘not there at all’. It is not possible to access an object that is still uploading.

    If you wish to ‘stream’ an object, then you might consider using a service that specifically streams, such as Amazon Kinesis.

    Login or Signup to reply.
  2. I don’t know if this fits your use case, but I use a Service Worker for this.

    It intercepts PUT requests to S3 and caches the resource without its signed parameters.

    On GET, it matches the base URL (again, without the signed parameters) and returns it from cache if possible.

    This means that you can stream straight from cache without having to wait for that upload to S3 to finish. This is particularly useful if you’re uploading an entire list of objects, one-by-one. Or, if you’re offline and are enqueuing uploads to run when you’re back online later.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search