Question 172664 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Why does Dynaconf Object Access fail after loading config from S3 unless iterated over?

I’m encountering a strange issue while loading TOML configuration files from an AWS S3 bucket into a Dynaconf object in Python.

Here’s a simplified version of the code I’m using:

import os
import boto3
from dynaconf import Dynaconf

def load_settings(template_name: str) -> Dynaconf:
    s3 = boto3.client("s3")
    key = f"{template_name}.toml"
    obj = s3.get_object(Bucket="my_bucket", Key=key)
    toml_str = obj["Body"].read().decode("utf-8")

    temp_file = f"{template_name}.toml"

    # Write the TOML string to the temporary file

    with os.fdopen(fd, "w") as file:
        file.write(toml_str)

    settings = Dynaconf(
        envvar_prefix="DYNACONF",
        environments=True,
        settings_files=[temp_file]
    )
    
    # Iterating over the items
    for k, v in settings.items():
        print(k, v)

    # Now I can access the values
    print(settings.my_value)

    os.remove(temp_file)
    return settings

The problem arises when I try to directly access a value from the settings object (for example, settings.my_value) after loading the configuration from the S3 bucket. This direct access fails unless I previously iterate over the items in settings.

Expected behavior: I should be able to directly access a value from the settings object without first iterating over all the items.

Actual behavior: Direct access fails with an error message stating that the requested key doesn’t exist, unless I first iterate over the items in settings.

This is particularly puzzling because if I comment out the iteration over the items in settings, the print statement fails, stating that ‘my_value’ doesn’t exist. But, if I leave the iteration in place, the print statement succeeds.

Any ideas why this might be happening? Is there something about how Dynaconf loads or accesses data that I’m missing here? Any help would be greatly appreciated!

Update: even better, give me a guideline on what would be the proper way of loading a remote settings file.

Tags:

2

Answers


  1. 0

    Dynaconf object access issue might be due to lazy loading. You can use Dynaconf’s remote function to load settings from S3 and merge them with your existing settings. This way, you can directly access the values without iterating over the items first. I’ve provided below code you can check and let me know.

    from dynaconf import Dynaconf

settings = Dynaconf(
    envvar_prefix="DYNACONF",
    environments=True,
)

settings.load_remote("s3://my-bucket/my-settings.toml")

# Now you can access the settings.my_value below
print(settings.my_value)
    Login or Signup to reply.
  2. 0

    The issue you’re encountering with Dynaconf is likely related to the order of operations when loading the configuration files. By default, Dynaconf loads settings lazily, meaning that the configuration values are not actually loaded until they are accessed or iterated over. In your code, when you iterate over the items in settings, it triggers the loading of the configuration values from the TOML file. This is why you can successfully access settings.my_value after the iteration. However, when you comment out the iteration, the configuration values are not loaded, leading to the error message stating that the key doesn’t exist. To ensure that the configuration values are loaded before accessing them, you can modify your code as follows:

    import os
import boto3
from dynaconf import Dynaconf

def load_settings(template_name: str) -> Dynaconf:
    s3 = boto3.client("s3")
    key = f"{template_name}.toml"
    obj = s3.get_object(Bucket="my_bucket", Key=key)
    toml_str = obj["Body"].read().decode("utf-8")

    temp_file = f"{template_name}.toml"

    # Write the TOML string to the temporary file
    with open(temp_file, "w") as file:
        file.write(toml_str)

    # Load the settings from the temporary file
    settings = Dynaconf(
        envvar_prefix="DYNACONF",
        environments=True,
        settings_files=[temp_file]
    )

    # Ensure that the configuration values are loaded
    settings.load()

    # Now you can access the values
    print(settings.my_value)

    os.remove(temp_file)
    return settings

    In the updated code, after creating the Dynaconf object, we explicitly call settings.load() to ensure that the configuration values are loaded before accessing them. This way, you don’t have to rely on iterating over the items to trigger the loading. Regarding the proper way of loading a remote settings file, the approach you’ve taken is reasonable. You fetch the TOML file from AWS S3, save it locally as a temporary file, and then load it into Dynaconf. Just make sure to handle any necessary error checking and exception handling for the S3 operations and file handling. Remember to adjust the code as needed for error handling and any additional logic specific to your use case.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search