I have a scenario where I get the last modified file from a specific S3 folder and there by I want to COPY the csv data into a redshift table. Below is the code:
def lambda_handler(event, context):
s3_client = boto3.client("s3")
list_of_s3_objs = s3_client.list_objects_v2(Bucket="demo", Prefix="myfolder/")
# Returns a bunch of json
contents = list_of_s3_objs["Contents"]
#get last modified
sorted_contents = sorted(list_of_s3_objs['Contents'], key=lambda d: d['LastModified'], reverse=True)
recent_file_uploaded = 's3://demo/'+sorted_contents[0].get('Key')
This yields output like – ‘s3://demo/myfolder/myfile.csv’
Next, I want to execute COPY command on this file. Below is the code:
#def redshift():
print('Redshift connection')
conn = psycopg2.connect(host="myhostname",port='5439',database="mydb", user=username, password=passw, sslmode='require')
print('connection success')
cur = conn.cursor();
# Begin your transaction
cur.execute("begin;")
print('Begin transaction')
cur.execute("copy mytable from recent_file_uploaded credentials 'aws_access_key_id='ACCESS_KEY';aws_secret_access_key='SECRET_KEY'' csv;")
# Commit your transaction
cur.execute("commit;")
print("Copy executed fine!")
In the above code, the cur.execute is failing with a syntax error at
copy mytable from recent_file_uploaded credential...
^
Not sure what’s wrong here. I tried making it as copy mytable from ''recent_file_uploaded''...
still same error. Can someone point me on where the COPY command is wrong pls?
2
Answers
Alright I am able to fix it with below syntax:
this fixed the problem. It's the way python parameters are passed into the COPY command.
Authorization parameters – Amazon Redshift shows an example using an Access Key and Secret Key as:
It would seem like you do not need to quote the individual elements. Admittedly, the documentation is confusing because it does show some examples with quotes, but they are not using credentials-args syntax.