I have PostgreSql 15.3 running as a docker container. My docker run configuration is -m 512g --memory-swap 512g --shm-size=16g Using this configuration, I loaded 36B rows, taking up about 30T between tables and indexes. I loaded with about 24 parallel…
I have a script like this for epoch in range(num_epochs): for bag in range(num_bags): feats = pd.read_csv(f"feats_{bag}.csv") ... # some logic As you can see, it repetitively reads data from a set of files. Each "feats_{bag}.csv" file is repetitively being…
I am creating a data ingestion pipeline using Cloud Run. My Cloud Run api gets called everytime a file is dropped in a GCS bucket via Pub Sub. I need to load some metadata that contains text for the data…
I cannot seem to access any shared memory functions. I'm running PHP7.1, provisioned using Easy Apache 4. From what I can make of the documentation shmop should just be there by default. I don't see any specific extension options in…