Question 103886 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – Container-based function app lost costom appsettings after being recycled by CORSSettingsChanged and AppSettingsChange

Our team has set up an ansible-playbook to deploy Azure Function as the reference: Linux-container-from-ACR-with-private-endpoint. We use a user-assigned identity to pull the function app image from an ACR. We automated the task using Ansible like the following:

- name: Force Login using identity
  command: az login --identity

- name: Get function app identity's clientId
  command: az identity show --resource-group "{{ functionapp_resource_group }}" --name "{{ functionapp_user_identity }}" --query clientId --output tsv
  register: identity

- name: Config properties
  command:
    az resource update --resource-group "{{ functionapp_resource_group }}" --name "{{ functionapp_name }}/config/web"
    --set "{{ item }}" --resource-type "Microsoft.Web/sites/config"
  with_items:
    - "properties.acrUserManagedIdentityID={{ identityId.stdout }}"
    - "properties.acrUseManagedIdentityCreds=true"

- name: Config app settings
  command:
    az function app config appsettings set --name "{{ functionapp_name }}"
    --resource-group "{{ functionapp_resource_group }}" --settings "{{ item }}"
  with_items:
    - "WEBSITES_ENABLE_APP_SERVICE_STORAGE=false"
    - "SubscriptionExpiredSchedule=* 30 * * * *"
    - "KEY=VALUE..."

- name: Deploy the function app
  command:
    az function app config container set --name "{{ functionapp_name }}"
    --resource-group "{{ functionapp_resource_group }}"
    --docker-custom-image-name "{{ docker_registry_host }}/{{ image_name }}:{{ image_tag }}"
    --docker-registry-server-URL "{{ docker_registry_host }}"

Everything looks fine when it just deployed. 2 timer-triggered functions are found correctly. The crontab string "* 30 * * * *" is detected from my app settings successfully:

enter image description here

But after a while, the log from Application Insight keeps showing 1 functions found and 0 functions loaded:

enter image description here

2023-01-13T10:15:47.434 [Information] 1 functions found
2023-01-13T10:15:47.434 [Information] 0 functions loaded
2023-01-13T10:15:48.352 [Information] Host Status: {
  "id": "<<function app name>>",
  "state": "Running",
  "version": "4.14.0.0",
  "versionDetails": "4.14.0+fc14864b4fc2a095f8cd92c7f0ca338dbeabd430",
  "platformVersion": "99.0.10.762",
  "instanceId": "b42e8886acdb8a5a6fd64d57ead538cc2765d75cbbc61a37cab2678ae174bea0",
  "computerName": "lw1sdlwk000014",
  "processUptime": 1677391,
  "functionAppContentEditingState": "Unknown",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "3.17.0"
  }
}

The function app was down. Back to Configuration, all of my custom appsettings lost either:

enter image description here

Although it kept failing by CI tool for the first time, it works fine after I trigger the CI tool manually for the second time and didn’t fail again. This is so weird. Did I set up any config incorrectly so the function app was down and the appsettings lost for the first time deployment?

— Update —

I found that the function app was redeployed because of Recycling container because of CORSSettingsChanged and CORSSettingsChanged = Microsoft.Web.Hosting.CorsSettings. After this deployment, all of my custom configs were lost. Besides, I found that Recycling container because of AppSettingsChange and isMainSite = True causes my custom configs lost as well. Here’s the log:

2023-01-17T02:29:27.044Z INFO - Pull Image successful, Time taken: 1 Minutes and 7 Seconds
2023-01-17T02:29:27.485Z INFO - Starting container for site
2023-01-17T02:29:27.485Z INFO - docker run -d --expose=80 --name <<functionapp name>>_0_4b241f6f -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=<<functionapp name>> -e WEBSITE_AUTH_ENABLED=False -e PORT=80 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=<<functionapp name>>.azurewebsites.net -e WEBSITE_INSTANCE_ID=5b6600aecfae7b5eb1290df18fcc42fb8bc88d7b721b9274087658bfa90a5d85 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False <<my acr>>.azurecr.io/<<name>>:<<tag>>

2023-01-17T02:29:27.485Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2023-01-17T02:29:29.726Z INFO - Initiating warmup request to container <<functionapp name>>_0_4b241f6f_msiProxy for site <<functionapp name>>
2023-01-17T02:29:29.874Z INFO - Container <<functionapp name>>_0_4b241f6f_msiProxy for site <<functionapp name>> initialized successfully and is ready to serve requests.
2023-01-17T02:29:29.874Z INFO - Initiating warmup request to container <<functionapp name>>_0_4b241f6f for site <<functionapp name>>
2023-01-17T02:29:39.659Z INFO - Container <<functionapp name>>_0_4b241f6f for site <<functionapp name>> initialized successfully and is ready to serve requests.
2023-01-17T02:37:22.159Z INFO - Recycling container because of CORSSettingsChanged and CORSSettingsChanged = Microsoft.Web.Hosting.CorsSettings
2023-01-17T02:37:28.816Z INFO - Pulling image: <<my acr>>.azurecr.io/<<name>>:<<tag>>
2023-01-17T02:37:29.124Z INFO - <<tag>> Pulling from <<name>>
2023-01-17T02:37:29.126Z INFO - Digest: sha256:49a690b289cd8a80d7104891a3ae5db38b8d6529f73b9b536c147d81a94a499e
2023-01-17T02:37:29.126Z INFO - Status: Image is up to date for <<my acr>>.azurecr.io/<<name>>:<<tag>>
2023-01-17T02:37:29.129Z INFO - Pull Image successful, Time taken: 0 Minutes and 0 Seconds
2023-01-17T02:37:29.188Z INFO - Starting container for site
2023-01-17T02:37:29.189Z INFO - docker run -d --expose=80 --name <<functionapp name>>_1_bf06fa66 -e WEBSITE_SITE_NAME=<<functionapp name>> -e WEBSITE_AUTH_ENABLED=False -e PORT=80 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=<<functionapp name>>.azurewebsites.net -e WEBSITE_INSTANCE_ID=5b6600aecfae7b5eb1290df18fcc42fb8bc88d7b721b9274087658bfa90a5d85 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False <<my acr>>.azurecr.io/<<name>>:<<tag>>

2023-01-17T02:37:29.189Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2023-01-17T02:37:31.739Z INFO - Initiating warmup request to container <<functionapp name>>_1_bf06fa66_msiProxy for site <<functionapp name>>
2023-01-17T02:37:31.819Z INFO - Container <<functionapp name>>_1_bf06fa66_msiProxy for site <<functionapp name>> initialized successfully and is ready to serve requests.
2023-01-17T02:37:31.820Z INFO - Initiating warmup request to container <<functionapp name>>_1_bf06fa66 for site <<functionapp name>>
2023-01-17T02:37:40.895Z INFO - Container <<functionapp name>>_1_bf06fa66 for site <<functionapp name>> initialized successfully and is ready to serve requests.

— Update —

I found that not only container deployment has this problem, but this problem still occurs after I switch to zip deployment. So it might be related to the outdated Terraform version (v1.0.4), not Docker agent on Azure. Also, my function app uses dependency injection, but I think it’s unlikely to have anything to do with it. Here’s the Terrafrom script:

resource "azurerm_app_service_plan" "B1" {
  name                = "plan-B1"
  resource_group_name = azurerm_resource_group.mygroup.name
  location            = azurerm_resource_group.mygroup.location
  kind                = "Linux"
  reserved            = true
  sku {
    tier = "Basic"
    size = "B1"
  }
}

resource "azurerm_function_app" "myapp" {
  name                = format("myapp-%s", terraform.workspace)
  resource_group_name = azurerm_resource_group.mygroup.name
  location            = azurerm_resource_group.mygroup.location
  os_type = "linux"
  version = "~4"

  storage_account_name       = azurerm_storage_account.myapp.name
  storage_account_access_key = azurerm_storage_account.myapp.primary_access_key

  app_service_plan_id = azurerm_app_service_plan.B1.id

  identity {
    type = "SystemAssigned"
  }

  app_settings = {
    APPINSIGHTS_INSTRUMENTATIONKEY = "${azurerm_application_insights.myapp.instrumentation_key}"
    FUNCTIONS_WORKER_RUNTIME = "dotnet"
  }

  site_config {
    linux_fx_version = "dotnet|6"
    always_on = true
    cors {
      allowed_origins = []
      support_credentials = false
    }
  }
}

In Addition, I found there is a mechanism called configuration drift, which enforces the resources to keep the same states as in Terraform. Since our appsettings depend on the environments, we use Ansible to set up each appsetting that definitely change the states. I highly guess this is the root cause of this issue.

I also opened a topic in Terraform’s forum: https://discuss.hashicorp.com/t/is-appsettings-disappearance-of-azure-function-relates-to-configuration-drift

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I finally figured out that it's because Azure forces my appsettings consistent with the terraform script. I solved it by adding ignore_change to allow changing app settings by Ansible.

    lifecycle {
  ignore_changes = [
    app_settings # prevent TF reporting configuration drift after app code is deployed
  ]
}

  2. 0

    Based on the exception Recycling container because of AppSettingsChange and isMainSite = True and in the Context of Docker Azure Function App, I found few existing resolved issues with the similar exception and context from the users @mpseidel and @Piotr that might helpful to the issue investigation analysis.

    First case in the GitHub-Azure-Functions-Host 3500 with the similar exception was coming at starting on Docker Linux based Node JS Azure Function App but that runs successfully locally and found the resolution as choosing the OS as Docker and got it worked.

    Second Case with the similar exception in the Context of Azure Linux based Function-Docker deployment SO #61567697.

    I can see that you have raised the triage ticket in GitHub Official Page of Azure Functions Issues and if any of the community members facing similar issue, here is the ticket-# 2327 labelled as bug and waiting for the triage.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search