Can not access secret from Azure Function Configuration using @Microsoft.KeyVault Reference through User-Assigned MSI

I have a bunch of different function apps.

They all need to access one secret from one key vault.

Because of that, I created a dedicated Managed Service Identity, and granted that MSI the ability to read the key vault.

Then, I add that MSI as a "user-assigned identity" of all the function app.

And then, inside configuration of function app, I established the following as the value of a configuration:

@Microsoft.KeyVault(SecretUri=https://key-vault-name.vault.azure.net/secrets/nameOfSecrets)

However, it seems like I do not have access to this key vault, even though I confirmed from the key vault home page that my user-assigned MSI do have read access to the key vault.

This is what the Configuration Source looks like:

and when I click it, it seems like it is still using the System Assigned Identity instead:

Any help is much appreciated.

I have read https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references?tabs=azure-cli , it says that I need to:

Configure the app to use this identity for key vault reference operations by setting the keyVaultReferenceIdentity property to the resource ID of the user-assigned identity.

But I don’t know how this can be done in the configuration of function app page.

I do not want to switch to System Assigned Identity. I do have a secret that’s from System Assigned Identity, but that one is different across different function apps. This one is the same across all function app, and the number of function app I have is not a constant number, adding them one by one to the key vault is just unmanageable.