I am getting below error when deploying Synapse CI/CD pipeline using Azure DevOps. The deployment was working fine until I added a few more notebooks and it started failing. Notebooks run fine on lower environment. All paths in the notebooks exist in higher environment. There is no concept of linkedTemplates in Synapse. I thought it was failing due to the length of ARMtemplate has exceeded.
Below is the error log, I have removed subscription name and resource group name and pipeline names.
2022-07-06T01:53:27.9739484Z Error during execution: Error: Failed to fetch the deployment status {"code":"400","message":"CreateOrUpdateNotebook failed: [statusCode from ADF:BadRequest, ErrorMessage: {"code":"BadRequest","message":"The document creation or update failed because of invalid reference ''.","target":"/subscriptions//resourceGroups/rg-name/providers/Microsoft.Synapse/workspaces/workspace-02/notebooks/SilverToGoldPipeline","details":null,"error":null}, workspace: workspace-02, notebook: SilverToGoldPipeline, ArtifactId: ]"}
2022-07-06T01:53:27.9811549Z ##[error]Encountered with exception:Error: Failed to fetch the deployment status {"code":"400","message":"CreateOrUpdateNotebook failed: [statusCode from ADF:BadRequest, ErrorMessage: {"code":"BadRequest","message":"The document creation or update failed because of invalid reference ''.","target":"/subscriptions/<>/resourceGroups/rg-name/providers/Microsoft.Synapse/workspaces/workspace-02/notebooks/SilverToGoldPipeline","details":null,"error":null}, workspace: workspace-02, notebook: SilverToGoldPipeline, ArtifactId: ]"}
2022-07-06T01:53:27.9954864Z ##[section]Finishing: Synapse deployment task for workspace: Stage
2
Answers
I was able to resolve the issue. The issue was that parameters file for higher environment was missing values for the pipelines. When we publish from master/main branch, if a new notebook is added and it is parameterized then an entry is created in the parameters file.
In my case, the error was because I had to create the Apache Spark pool first, before deploying.
Apparently, the deployment was not creating it automatically.