Original post I want to try out the Apache Arrow Flight SQL Driver for a large OLAP query on a PostgreSQL database. When I run the following example: import adbc_driver_flightsql.dbapi import adbc_driver_manager conn = adbc_driver_flightsql.dbapi.connect('flightsql://username:[email protected]:5432/database') with conn.cursor() as cur: cur.execute("SELECT…
I want to access Parquet files on an Azure data lake, and only retrieve some rows. Here is a reproducible example, using a public dataset: import pyarrow.dataset as ds from adlfs import AzureBlobFileSystem abfs_public = AzureBlobFileSystem( account_name="azureopendatastorage") dataset_public = ds.dataset('az://nyctlc/yellow/puYear=2010/puMonth=1/part-00000-tid-8898858832658823408-a1de80bd-eed3-4d11-b9d4-fa74bfbd47bc-426339-18.c000.snappy.parquet',…
Please have a look at the reprex at the end of the post. I need to read a column as a string, perform several manipulations and then save convert it to a numerical column. The blanks ("") in the string…