skip to Main Content

I have a scenario where I get the last modified file from a specific S3 folder and there by I want to COPY the csv data into a redshift table. Below is the code:

def lambda_handler(event, context):
    
    s3_client = boto3.client("s3")
    list_of_s3_objs = s3_client.list_objects_v2(Bucket="demo", Prefix="myfolder/")
    
    # Returns a bunch of json
    contents = list_of_s3_objs["Contents"]
    
    #get last modified
    sorted_contents = sorted(list_of_s3_objs['Contents'], key=lambda d: d['LastModified'], reverse=True)

    recent_file_uploaded = 's3://demo/'+sorted_contents[0].get('Key')

This yields output like – ‘s3://demo/myfolder/myfile.csv’

Next, I want to execute COPY command on this file. Below is the code:

#def redshift():
    print('Redshift connection')
    conn = psycopg2.connect(host="myhostname",port='5439',database="mydb", user=username, password=passw, sslmode='require')
    print('connection success')
    
    cur = conn.cursor();

    # Begin your transaction
    cur.execute("begin;")
    print('Begin transaction')
    
    cur.execute("copy mytable from recent_file_uploaded credentials 'aws_access_key_id='ACCESS_KEY';aws_secret_access_key='SECRET_KEY'' csv;")
    # Commit your transaction
    cur.execute("commit;")
    print("Copy executed fine!")

In the above code, the cur.execute is failing with a syntax error at

copy mytable from recent_file_uploaded credential...
                  ^

Not sure what’s wrong here. I tried making it as copy mytable from ''recent_file_uploaded''... still same error. Can someone point me on where the COPY command is wrong pls?

2

Answers


  1. Chosen as BEST ANSWER

    Alright I am able to fix it with below syntax:

      copy_query = f"COPY mytable FROM '{recent_file_uploaded}' IAM_ROLE 'my-role-arn' FILLRECORD CSV;"
        cur.execute(copy_query)
    

    this fixed the problem. It's the way python parameters are passed into the COPY command.


  2. Authorization parameters – Amazon Redshift shows an example using an Access Key and Secret Key as:

    CREDENTIALS 'aws_access_key_id=<temporary-access-key-id>;aws_secret_access_key=<temporary-secret-access-key>;token=<temporary-token>'
    

    It would seem like you do not need to quote the individual elements. Admittedly, the documentation is confusing because it does show some examples with quotes, but they are not using credentials-args syntax.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search