Read "Integrated Dataset" into a Dataframe with Azure Synapse Notebook
I know how to read a file in the Azure Data Lake Storage, but I don't know how to read a file that is listed under the Integration Dataset section in Azure.
Azure – Issue with synapse workspace creation
enter image description here Above is the screenshot for the error. {"code":"DeploymentFailed","target":"/subscriptions/xyz/resourceGroups/iam/providers/Microsoft.Resources/deployments/Microsoft.Azure.SynapseAnalytics","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"ReachedPerSubscriptionWorkspaceLimit","message":"Reached the maximum number of Synapse workspaces allowed for this subscription.…
Azure – Synapse Pipeline Trigger parameters
We have synapse pipeline that works fine if manually triggered . But when ran in a schedule has the following error message and fails . I verified the parameter names and the parameter does exist in the definition . Any…
Azure Synapse — Copy Activity — run a query that pulls from tables in more than one database?
I'm looking to use Azure Synapse's Copy activity to pull data in a big select that has tables coming from more than a single database. Something like this in traditional SQL using Linked Servers -- Select t1.fielda, t1.fieldb, t2.fieldc, t2.fieldd…
Azure – How to create an external table using variables in Synapse without specifying columns
I have a stored procedure in Synapse Dedicated pool which takes a parquet file from adls and creates a staging table in dedicated pool: ALTER PROCEDURE dbo.Staging_Tables_sp ( @TableName VARCHAR(MAX), @SchemaName VARCHAR(MAX) ) AS BEGIN EXEC ('SET ANSI_NULLS ON; SET…
Azure – Can't insert external table data into table in synapse dedicated sql pool
In Synapse dedicated pool, I'm trying to create an external table, and a dedicated table, and then insert the external table into the dedicated table but I keep getting the following error: Explicit conversion from data type bigint to date…
Azure – ADF Pipeline to Succeed even if activity fails
I have a pipeline which i'm working on, i'm doing a lookup to check if few files are there, if the folder path is not there i have to perform certain processes, when the folder path is not present, the…
Azure – How can we encrypt specific column data of parquet using pyspark
There is a requirement in my project to encrypt some of PII columns data while writing data in a parquet file. To write the data in parquet file, Azure Synapse pyspark notebook is being used. Not getting any references on…
Azure – Data Factory – pass json to foreach activity
In DataFactory, I'm creating a ForEach activity and in Items I'm using this expression: @createArray(json('{ "daily": { "raw": [ { "entity_name": "ent1", "fetchXML_file": "raw/ent1.xml", "destination_path_base": "/master/raw/ent1" }, { "entity_name": "ent2", "fetchXML_file": "raw/ent2.xml", "destination_path_base": "/master/raw/ent2" }, { "entity_name": "ent3", "fetchXML_file": "raw/ent3.xml",…
Is it possible to create a date array variable dynamically in Azure Synapse Analytics?
In Azure Synapse Analytics, I would like to use the pipeline function to set a variable of type array for the date from 12 months ago to current date. The date in the variable is the first day of each…