I want to upload only Ec2 log files through lambda.
I have code for this
import json
import os
import datetime as dt
import boto3
import socket
region = 'ap-northeast-2'
instances = []
ec2_r = boto3.resource('ec2')
ec2 = boto3.client('ec2', region_name=region)
for instance in ec2_r.instances.all():
for tag in instance.tags:
if tag['Key'] == 'Log-Archive':
if tag['Value'] == 'True':
instances.append(instance.id)
def lambda_handler(event, context):
s3 = boto3.client("s3")
dir_path = "/log"
bucket = 'mybucket'
x = dt.datetime.now()
date = x.strftime("%Y%m%d")
def log(in_path):
for (dir_path, dir, files) in os.walk(in_path):
for file in files:
if date in file:
yield os.path.join(dir_path, file)
for file_name in log(dir_path):
key = socket.gethostname() + '/' + file_name
res = s3.upload_file(file_name, bucket, key)
return {
'statusCode': 200,
}
When I tested this code it returned 200 code. but nothing in my bucket.
And I tested this code on my local ec2 without lambda code, and It works
33
Response
{
"statusCode": 200
}
Function Logs
START RequestId: 39f63d4f-e154-48b8-9fb3-51b98f927a30 Version: $LATEST
END RequestId: 39f63d4f-e154-48b8-9fb3-51b98f927a30
REPORT RequestId: 39f63d4f-e154-48b8-9fb3-51b98f927a30 Duration: 51.61 ms Billed Duration: 52 ms Memory Size: 128 MB Max Memory Used: 89 MB
Request ID
39f63d4f-e154-48b8-9fb3-51b98f927a30
Can I take some advices? thanks for all answers.
2
Answers
Your Lambda will be running in a serverless compute environment, not in your EC2. That means the
log
directory that you are traversing is in the serverless Lambda environment, not thelog
directory in your EC2 instance. My guess is, the Lambdalog
directory contains no log files. Hence, the code ran successfully without uploading anything to S3.My suggestion is to keep this python as a local script in the instance. Then, as you have already indicated, you can use AWS SSM to trigger this script.
you probably miss to install in your ec2 the cloudwatch agent.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
Furthermore, there is no need to "move" log between services, because AWS provide automatic logging for everything, with Cloudwatch and Cloudtrail services