Question 102292 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – How to send CSV file from an api endpoint to S3 bucket in node js lambda

I have this particular scenario where I am calling an external API which returns csv file as response, now my requirement is to send this CSV file to S3 bucket on the fly, this means without downloading the file to temporary storage, I want to send it to S3.

I am writing lambda using node JS.
Sample pieces of code to do this would be helpful

Thanks

Tags:

2

Answers


  1. 0

    The lambda function looks like follows:

    const AWS = require('aws-sdk');
const axios = require('axios');

const apiUrl = 'https://sample-videos.com/csv/Sample-Spreadsheet-100-rows.csv';

function putObjectToS3(bucket, key, data){
    var s3 = new AWS.S3();
    var params = {
        Bucket: bucket,
        Key: key,
        Body: data
    }
    s3.putObject(params, function(err, data) {
      if (err) console.log(err, err.stack);
      else     console.log(data);
    });
}

exports.handler = async function(event, context) {
  const response = await axios.get(apiUrl);
  console.log(response.data);
  putObjectToS3('bucket-name-goes-here', 'file-name.csv', response.data);
  return context.logStreamName;
};

    NOTE: Remember to add IAM permission to the AWS Lambda Role under policies

    {
    "Sid": "AnyNameForThePermission",
    "Effect": "Allow",
    "Action": "s3:*",
    "Resource": [
        "arn:aws:s3:::bucket-name-goes-here/*"
    ]
}

    PREREQUISITE: Since Axios is not present by default in Lambda, create a layer for it or add it to an existing layer. Check the link for more details.

    Login or Signup to reply.
  2. 0

    You will need to download the CSV file to local storage. The S3 API supports copyObject() which is for copying from one s3 location to another s3 location. But if the location of the source CSV is not in s3, then you’ll need to download the CSV file to the local storage of the Lambda first.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search