Introduction The script I am writing reads from a .csv file that has the format of schema, table, column, and then executes a SELECT query to get the values of all records of those columns. My goal with the script…
I am making a data pipeline using python and am pulling a 3gb table from our RDS instance using psycopg2. Im currently testing this locally, and whenever I try this my Ram says my python instance is taking up 27gb…
I'm wondering how I would parameterise the following postgres: UPDATE your_table as t SET val = x.val_to FROM ( VALUES (1, 20), (3, 44) ) as x (val_from, val_to) WHERE t.val = x.val_from AND your_table.other_id = %(other_id_value) ; where (1,…
I have a table myuser with two columns (id serial primary, username varchar(20) unique not null). I am trying to make a Tkinter gui that takes "username" from a form and inserts it in the postgres table myuser. I want…
I am using extras.execute_values() method in psycopg2 and things don't go throw as I expected: countyid_landid_tuple_list = [(1,1),(1,2),(2,3)] sql = "update agri_land set county_id = %s" extras.execute_values(cur, sql, countyid_landid_tuple_list ,"%s where id = %s") The error message is as follows:…
I am trying to run the following query on an SQL table of about 2.66 billion rows, keep in mind I am using Python psycopg (but I can use something else if it would make my life easier) to eventually…
This code is reading data in chunks between certain dates. The problem is - once I stop the script, for instance with ctrl-c, and then restart, the cursor position continues at the same position / datetime where it stopped. How…
I'm trying to build a stack using AWS CDK, which includes an RDS PostgreSQL instance and a Lambda function. From the Lambda, I'm trying to access the PostgreSQL, but I'm not succeeding in any way. My problem is that no…
i made a project on django. i work in virtualenv, i installed psycopg2-binary, tried to connect postgresql instead of sqlite3. rewrite code in settings.py DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'NAME': 'database', 'USER': 'user', 'PASSWORD': 'mypassword', 'HOST': 'localhost', 'PORT':…
I have a list of Python dictionaries that look something like below where each dictionary can have slightly different keys. data = [ {'name': 'Bob', 'age': 32}, {'name': 'Sara', 'city': 'Dallas'}, {'name': 'John', 'age': 45, 'city': 'Atlanta'} ] I also…