I have a DF column which is a long strings with comma separated values, like: 2000,2001,2002:a,2003=b,2004,100,101,500,20 101,102,20 What I want to do is to create a new Array<Int> column out of it where: only values starting with 2 are included…
I have been trying to find a Cron expression to use in my AWS Glue Job. I tried many examples and I'm still not sure if this is possible or NOT, since this is the first time I'm adding time…
What is the best way to explode a comma separated column based on specific conditions? I have some data in the following format: ID col1 1 a100,a101,b100,c100 2 a105,b100 3 b101, c104 what I want to achieve is to: grab…
I have downloaded the AWS glue script from my Dev environment by clicking "Download" as a JSON file which include Job parameter and other info as well other than the code. When i upload to my production environment by clicking…
Data has headers like _col_0, _col_1, etc. I have a sample data file that has the correct column headers. However, all the data is in snappy/parquet across ~250 files. What is the easiest way to remap the column headers in…
I'm a little bit desperate to automate the AVRO schema registration for AWS Glue Schema Registry. According to the official documentation (https://docs.aws.amazon.com/glue/latest/dg/schema-registry-gs.html#schema-registry-gs4) to register a new AVRO schema following command must be executed: aws glue create-schema --registry-id RegistryName="my-registry-name" --schema-name testschema…
I have this schema in the AWS Glue job: root |-- SortedLenders: array | |-- element: struct | | |-- LenderID: string | | |-- MaxProfit: string |-- FilteredOutDecisions: array | |-- element: struct | | |-- ApprovedAmount: string |…
I currently have a DynamoDB stream configured which is inputing streams into Kinesis Data streams whenever insertion/updation happens and subsequently I have Glue tables which is taking input from above kinesis stream and then displaying the structural schema and also…
As in the title of my post, it is possible? I don't see any options. Also, the job details include advanced properties that include the name of my script, for example, job-name.py When the python script is renamed, will the…
I want to read a JSON file in PySpark, but the JSON file is in this format (without comma and square brackets): {"id": 1, "name": "jhon"} {"id": 2, "name": "bryan"} {"id": 3, "name": "jane"} Is there an easy way to…