The configBuilder 'AzureKeyVault' failed while processing the configuration section 'appSettings'

Thanks to @Harshitha for pointing me in the right path, using connected services in VS 2019 to connect to a keyVault which can then ref values using appSettings.

To test this I created a new dummy app using a .Net 4.8 framwork application in C#

I have followed this clip:
https://www.youtube.com/watch?v=S7EPrlpPqXw

Basically, use connected services to connect to your key vault.

This will include the following in your web.config file:

 <configuration>
      <configSections>
        <section name="configBuilders" type="System.Configuration.ConfigurationBuildersSection, System.Configuration, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" restartOnExternalChanges="false" requirePermission="false" />
      </configSections>
      <configBuilders>
        <builders>
          <add name="AzureKeyVault" vaultName="RealKeyVaultName" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
        </builders>
      </configBuilders>
      <appSettings configBuilders="AzureKeyVault">
       <!-- Value added by me -->
        <add key="secretInKV" value="dummyValue" />
     </appSettings>
    </configuration>

So basically creating a connection to KV using configSection and configBuilders

In code I can then say

var secretValue = ConfigurationManager.AppSettings["secretInKV"];

and this correctly returns the value stored in my KV, (not dummyValue from the above app settings) which is all working fine.

However when I try to add this to my real application I get an error loading:

Parser Error Message: The configBuilder ‘AzureKeyVault’ failed while processing the configuration section ‘appSettings’.: Error in Configuration Builder ‘AzureKeyVault’::GetValue(secretInKV)

The stack trace errors show:

[SocketException (0x2746): An existing connection was forcibly closed by the remote host]
[IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.]
[WebException: The underlying connection was closed: An unexpected error occurred on a send.]
[RequestFailedException: The underlying connection was closed: An unexpected error occurred on a send.]
[AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy.]
[Exception: Error in Configuration Builder 'AzureKeyVault'::GetValue(secretInKV)]

To test my connection to KV In code I can say:

var client = new SecretClient(new Uri(keyVaultURL), new DefaultAzureCredential());
var secret = client.GetSecret(secretInKV);

and as all of the depenedencies where added when using the connected service I am able to retrieve the value from KV, but I want to get it from app settings

if I remove

configBuilders="AzureKeyVault"

from

 <appSettings configBuilders="AzureKeyVault">

the application loads, why is this causing an issue please?

I have read similar posts online but was not able to solve,
I am properly connected else I wouldnt be able to get the value with the above mentioned C# code, so why is this causing an issue please?
thank you for any replies

I have matched the Nuget packages in the new dummy app I have created against my actual application and still this issue is happening