Question 104924 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – Alternative for BlobContainerClient.CreateIfNotExistsAsync method to avoid 409 http error

Prerequisites

  • WindowsAzure.Storage V 9.3.3
  • Azure.Storage.Blobs V 12.6.0
  • Datadog APM monitoring

Context:

  1. This is a known problem, that a blob client CreateIfNotExists(Async) returns 409 during execution, in case if a container already has been created before
  2. Due to existed implementation, the CreateIfNotExists check is executing per a request, that mean a lot of times
  3. As a result, there are a lot of 409 errors in a log & monitor systems
  4. Also, it’s kind of challenge to move logic to the app root, and, thus, to avoid per request CreateIfNotExists check

The possible fix, under consideration:

To mitigate the issue, I gonna to replace it with something like:

if (!await blobClient.ExistsAsync().ConfigureAwait(false))
{
   await containerClient.CreateAsync().ConfigureAwait(false);
}

Question: And one moment that bothering me:

  • Could be such replacement become a concern within high concurrent workflow ?

P.s I’ve looked into CreateIfNotExistsAsync implementation at ‘Azure.Storage.Blobs‘ nuget. From my point of view, nothing special about concurrency, but maybe I’m wrong. Pls. share you experience

enter image description here

Tags:

3

Answers


  1. 0 
    BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);

            if (!containerClient.Exists())
            {
                containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName, PublicAccessType.BlobContainer);
            }
    Login or Signup to reply.
  2. 0

    Your code will have to handle this problem. Checking for existence won’t help.

    The problem is concurrency on the server side, not concurrency in the client. The source shows thatCretaeIfNotExists already handles the case of an existing container by simply ignoring the exception. One could ask why an exception is thrown instead of checking the status code, but the code doesn’t throw a 409 if a container exists:

    try
{
    response = await CreateInternal(
        publicAccessType,
        metadata,
        encryptionScopeOptions,
        async,
        cancellationToken,
        operationName)
        .ConfigureAwait(false);
}
catch (RequestFailedException storageRequestFailedException)
when (storageRequestFailedException.ErrorCode == BlobErrorCode.ContainerAlreadyExists)
{
    response = default;
}
catch (Exception ex)
{
    ClientConfiguration.Pipeline.LogException(ex);
    scope.Failed(ex);
    throw;
}
finally
{
    ClientConfiguration.Pipeline.LogMethodExit(nameof(BlobContainerClient));
    scope.Dispose();
}
return response;

    For a conflict to occur, two CreateInternal calls would have to try to create the same non-existent container. That’s why checking for existence won’t help here. Even if an existence check was used, both clients would still try to execute Create at the same time and both would get a 409.

    The application will have to handle this case. The best option would be reducing the chance of conflicts by not making this call on every request. This could be done by using eg a Lazy<> or asynchronous Lazy to make this call. Or using queued workers, reducing the number of calls that can happen at the same time.

    The application would still have to handle the occasional 409. One way would be to simply retry CreateIfNotExists after a delay, until the call succeeds. That’s better than calling Exists because the container’s creation could fail.

    Login or Signup to reply.
  3. 0

    I just wanted to chime in here because I suspect the cause for concern here is not what the MS is or is not doing, its what Datadog is doing as I just dealth with this issue this morning. The OP is using Datadog as their APM, another user mentioned AppInsights. At the end of the day the APM wouldn’t matter but the use of one will cause these false-positive errors. The APM is observing the 409 in the response from the blob storage service itself. Regardless of whether or not a library catches and eats it or not, the blob service is still returning a 409 and its response code is observed by the APM. The 409’s are being caught and handled by the MS lib and the 409 response to the attempt to create a container that already exists, as the call needs to be idempotent and cannot rely on checks due to concurrency concerns, is completely expected.

    The solution to this problem lies in the configuration of the APM, not in the application code. In our case, given that we have a blob persistence logic encapsulated in our own class that handles the container/blob interaction along with logging and other elements, was to configure the APM to ignore 409’s from a specific method within the class and moved the call to create the BlobContainerClient and CreateIfNotExistAsync to the new method now being conditionally ignored by the APM. This does not impact 409’s for other operations, including those dealing with the contents of the blobs and avoids the substantial number of 409’s which are returned while implementing the best practices of using CreateIfNotExist.

    Note: I think some responders might have overlooked the fact that it is very common to have a container, not blob, but a container, used to store related content in blobs. Just like you would use a folder to store related files. In our case, we had a container named "content" which has hundreds of thousands of files in it, which, if we wanted to implement custom logic or DevOps processes to pre-create them, we could and then skip the code to CreateIfNotExist but that is brittle. The 409’s in this use case are both expected and necessary outside of creating highly coupled code, try/catch/retry for first attempts or some external out of band process such as a DevOps pipeline. Microsoft was aware of this and provides a client to solve this common dilemma and for not the APM, the OP would not have known it was occurring.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search