Question 104978 in Azure Question posted in
The official documentation to get you started can be found here.

Azure Data Factory Dynamic content default values per environment

I hope somebody can help me understand how I can solve this issue.

In azure I have 3 separate environment:

  • Development
  • Staging
  • Production

Each of this environment I have a Datafactory. Due to the large datasets I have I decided to build a CI CD pipeline to copy all the content of dev environment to staging and production every time I publish to adf_publish. This process work like a charm and I have all the datasets and linked services.

Here is where I start finding issues and I couldn’t pass over them.

all 3 environments have cosmos databases with the exact same structure, but of course the url and access keys to cosmos are different.

Reading in Microsoft documentation, I have found that I can use dynamic content which is absolutely perfect.
I created the parameters:

enter image description here

when I try the connection in dev environment, the parameters blade opens and I can set the default values I want and the connection succeded.

I did the same configuration in staging and after the release pipeline, I could see the Parameters,

enter image description here

But as you can see from the screenshot nothing else has been configured. My first thought was that maybe I need to configure the default values and apply the changes but even after that, after each release the values get reseted.
Can anyone please help me understand how I can set the default values for each environment?

Please if there is any question or something I didn’t explained well, just ask me and I will rephrase.

Thank you very much for any help you can provide me with.

UPDATE:
Finally I managed to sort the first part of my problem, which is make the release pipeline set the dynamic content in the cosmos connectionenter image description here

But unfortunately my parameters value or they are hard coded or empty.

following this documentation https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment

I tried to create a arm-template-parameters-definition.json in the adf_publish in the root folder with this values:

{
"Microsoft.DataFactory/factories/linkedservices":{
    "properties": {
        "parameters": {
            "DBNameAggregated": {
                "type": "string",
                "defaultValue": "test"
            },
            "DBAccessKey": {
                "type": "string",
                "defaultValue": "test"
            }
        }
    }
}
}

when I saved and pushed the changes, I refreshed the portal and published.
in my release pipeline I set the override param

-DBNameAggregated "test2"

but the release pipeline fails with the following error

Deployment template validation failed: 'The template parameters 'DBNameAggregated' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time

I checked both my arm templates:

  • TemplateForFactory: Here I have the parameters declared.
  • TemplateParameters: Here the params are not declared

I am really hitting a wall here not knowing what I am doing wrong

Tags:

2

Answers


  1. 0

    You will need to create the arm-template-parameters-definition.json under the main root folder.

    arm-template-parameters-definition

    If you want to parameterize the values, you need to add the below code.

    In this example, I am parameterizing all values under typeProperties for cosmosdb linked service

    "Microsoft.DataFactory/factories/linkedServices": {
    "CosmosDB": {
        "properties": {
            "typeProperties": {
               "*": "="
            }
        }
    }
}

    Once you have updated this code, You will see the changes on the ARMTemplateforFactory.json file under the adf_publish branch.

    Here are the values from the armtemplateforfactory.json file.

    CosmosDb1_connectionString, cosmosdbURL, CosmosDBKey are parameterized.

    "name": "[concat(parameters('factoryName'), '/CosmosDb1')]",
        "type": "Microsoft.DataFactory/factories/linkedServices",
        "apiVersion": "2018-06-01",
        "properties": {
            "parameters": {
                "cosmosdbURL": {
                    "type": "string",
                    "defaultValue": "xyz"
                },
                "CosmosDBKey": {
                    "type": "string",
                    "defaultValue": "xyz"
                }
            },
            "annotations": [],
            "type": "CosmosDb",
            "typeProperties": {
                "connectionString": {
                    "type": "SecureString",
                    "value": "[parameters('CosmosDb1_connectionString')]"
                }
            }
        }

    My actual code on the cosmosdb linked connection. Here I have assigned xyz to the cosmosdburl and cosmosdbkey

    {
"name": "CosmosDb1",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
    "parameters": {
        "cosmosdbURL": {
            "type": "string",
            "defaultValue": "xyz"
        },
        "CosmosDBKey": {
            "type": "string",
            "defaultValue": "xyz"
        }
    },
    "annotations": [],
    "type": "CosmosDb",
    "typeProperties": {
        "connectionString": "AccountEndpoint=@{linkedService().cosmosdbURL};Database=bhargavademo;AccountKey=@{linkedService().CosmosDBKey}"
    }
    }
}

    The last step is, you need to update the cosmosDBURL, CosmosDBKey, connection string values on your overridetemplateparameters on the release pipeline.

    Override Template parameters

    Login or Signup to reply.
  2. 0

    Connection Strings are secure strings and will not have a default value (see here)

    In order to retain the value you need to override the template parameters inside of the ARM deployment step during your CI/CD pipeline. The parameter is called {your linked service name}_connectionstring. You can find all the parameters inside of the ARMTemplateParametersForFactory.json file.
    enter image description here

    For the default values of the linked service parameters (DBNameAggregated & DBAccessKey) follow the steps described in BhargavaGunnam-MSFT answer

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search