I'm trying to read a parquet folder using the following code: import pandas as pd df = pd.read_parquet('PASP0001.parquet') I'm working on a virtual environment. The code works perfectly if I open a Python session (note the which command): (.venv) (base)…
I am trying to upload a parquet file to Azure, but it keeps giving me an error message saying that its missing a required HTTP header. In the example I am providing I just tried some basic data in a…
I have a use case as follows: An IoT device is uploading many small files into an AWS S3 input bucket. Upon upload, every single file is processed by an AWS Lambda function to decode the data and write a…
There is a requirement in my project to encrypt some of PII columns data while writing data in a parquet file. To write the data in parquet file, Azure Synapse pyspark notebook is being used. Not getting any references on…
Using PySpark 2.4.7 My goal is to write PySpark DataFrame into specific number of parquet files into AWS S3. Say I want to write PySpark DataFrame into 10 parquet files. This is my approach df.repartition(10).write.mode("append").partitionBy(partition_cols).saveAsTable(<db.table>) This writes 10 parquet files…
I want to read the Parquet files I get from the readSteam() method from a Kafka topic. However, when I read these Parquet files, I get a dataframe that I don't want instead of real values. import glob parquet_files =…
I have Azure Data Factory, with Integration Runtime, with installed Java (OpenJDK), Parquet files in Azure Blob, and a destination table in SQL Server. I manage to get some fields read from Parquet and written into the SQL Server table…
Data has headers like _col_0, _col_1, etc. I have a sample data file that has the correct column headers. However, all the data is in snappy/parquet across ~250 files. What is the easiest way to remap the column headers in…
I'm starting a project to adjust the data lake for the specific purge of data, to comply with data privacy legislation. Basically the owner of the data opens a call requesting the deletion of records for a specific user, and…
I have Paraquet files in my S3 bucket which is not AWS S3. Is there a tool that connects to any S3 service (like Wasabi, Digital Ocean, MinIO), and allows me to query the Parquet files?