Loading config data from json file in AWS template.yaml for deploying to AWS Lambda – Amazon web services

Starting from the thread that iagotb shared, I updated my approach to tackle this issue: work with AWS parameters in template.yaml.

• Add the data that I want to insert as a parameter in template.yaml
• Refer to that parameter elsewhere in template.yaml
• Override that parameter's value during deployment using the --parameter-overrides flag of sam deploy

As the tools that I deploy are deployed automatically via a Jenkins environment, and I don't have control over that environment and what libraries are installed there, the suggested solutions in the linked thread didn't directly work for me. In the end I followed these steps to add the parameter overrides to my deployment:

• create build.py file that is called during Jenkins deployment
• automatically create .samparams from build.py
• load .samparams into variable during Jenkins deployment
• pass variable to sam deploy

Below I will explain in more detail what I did.

Original desired solution

What I originally wanted, was to directly plug data from a json file in my AWS template and format that data correctly in the template, for example by doing something like:

VERSION: ${file(config.json):version_number}

However, this is simply not possible.

Creating .samparams

During the Jenkins deployment, I automatically create .samparams from build.py. In this process, I can fully control all information sources (a json, another config file, or just hard-coded in build.py) to collect the necessary data to create the parameter overrides.

Next I create the file.samparams according to the following formatting rules:

• parameter name-value pairs are written as: parameter_name=parameter_value
• names and values are not escaped with double quotes
• multiple pairs are separated by spaces
• the whole string should be on 1 line

For example, let's say we have three parameters we want to override:

• "Version": "1.4"
• "Env": "prod"
• "AppName": "my_example_app" then a correct .samparams file contains the following line:

Version=1.4 Env=prod AppName=my_example_app

Passing .samparams to sam deploy

The Jenkins deployment is controlled via a Groovy file. In the deployment stage I execute the following two commands:

SAM_PARAMS = sh (script: "(cat ./.samparams)", returnStdout: true)

sh """
(
. ${PYTHON_VENV}/bin/activate;
sam deploy --no-confirm-changeset --parameter-overrides ${SAM_PARAMS}
)
""".stripIndent()

And with this, the parameter overrides are correctly passed to sam deploy.

NOTE: When supplying parameter overrides via the command line, AWS will ignore all existing parameter overrides in the samconfig.toml file. If you have any overrides listed there, make sure to merge them into .samparams. This is a limitation of AWS.

Alternative: update samconfig.toml

Prior to above solution, I explored the possibility to dynamically update an existing samconfig.toml file where other AWS configuration settings are stored. While this is more powerful, as you can define parameter overrides for each AWS command and environment, I abandoned that approach due to the complexity of working with the parameter overrides field and the (limited) capabilities of toml parsers in Python.

1. In samconfig.toml the parameter overrides should be in a similar format as above .samparams format. However the parameter values should be surrounded by double quotes and the double quoted should be escaped:

   parameter_overrides = "Version="1.4" Env="prod" AppName="my_example_app""
   

2. The toml parsers I tried are (were) quite finicky with the backslashes, which made it very cumbersome and difficult to create correct parameter overrides strings.
3. Toml is expressed as a dict in Python. So, if you want to create a field in a table that doesn't exist yet, you need to check and potentially create all intermediate tables before you can create the overrides field.

   For example, if you want to create

   config['default']['deploy']['parameters']['parameters_override']
   

   then you need to check whether default, deploy, and parameters exist (and create them if they don't) before you can assign a value to the parameter_overrides field. toml parsers don't do that for you.

4. Due to the deep nesting, your code becomes very verbose, very quickly.

Given all the above and that for my use case I only needed to supply parameter overrides to sam deploy, I abandoned this approach and went for the .samparams file approach instead.