Problem with downlowding very large files to Azure Blobstorage through Output stream via Flux<DataBuffer> and DataBufferUtils.write(

user286974
July 3, 2023
85 views
0 votes
2 Answers

I need to download very large file via REST and store it to an Azure Blobstorage. I am facing quite some Problems.

For all examples, I use this call to get to the Data

 var flux = this.webClient
            .get()
            .uri(urlToAssert)
            .accept(MediaType.APPLICATION_OCTET_STREAM)
            .exchangeToFlux(clientResponse ->                  
clientResponse.body(BodyExtractors.toDataBuffers()));

This code works for file with Size 50MB, 1GB, 5GB. Note is write directly the Files into the file system.

try (OutputStream outputStream = new FileOutputStream(targetFile.resolve("testData.zip").toAbsolutePath().toString(), false)) {
            DataBufferUtils.write(flux, outputStream).map(DataBufferUtils::release).blockLast(Duration.ofHours(22));
        } catch (Exception e) {
            throw new IllegalStateException(e);
        }

So i know the Stream handling is ok. Now i use AzureBlob Storage OutputStream. The 50mb works. The 1GB is also present in the Blobstorage, but the method keep hanging somewhere in the "try"

  try (OutputStream outputStream = this.blob.getOutputStreamParallel(destination, this.blob.getParallelTransferOptions())) {
            DataBufferUtils.write(flux, outputStream).map(DataBufferUtils::release).blockLast(Duration.ofHours(22));
            outputStream.flush();
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }

    public OutputStream getOutputStreamParallel(Path path, ParallelTransferOptions parallelTransferOptions) {
        var blobClientTarget = this.containerClient.getBlobClient(relativePathUnix(path));
        return blobClientTarget.getBlockBlobClient().getBlobOutputStream(parallelTransferOptions, null, null, null, null);
    }

this.parallelTransferOptions = new ParallelTransferOptions()
            .setBlockSizeLong(40 * Constants.MB)
            .setMaxConcurrency(5)
            .setProgressListener(bytesTransferred -> {
                log.info("write bytes, bytes transferred '{}'", bytesTransferred);
            });

In the log i see the following strange thing

Does somebody see my error? Or is the Azure BlobStorage broken?

Answers

Chosen as BEST ANSWER
- user286974
- July 3, 2023 at 10:05 am
- 0 votes
0
The Problem was that I used the BlobContainerClient and not the BlobContainerAsyncClient. The BlobContainerAsyncClient has special API's to handle the Flux-"Stuff"

Hear is the code that I use now:
```
 public void uploadAsync(Flux<DataBuffer> flux, Path destination) {
        BlobAsyncClient blobClientTarget = this.blobContainerAsyncClient.getBlobAsyncClient(relativePathUnix(destination));
        blobClientTarget.upload(flux.map((dataBuffer) -> {
            ByteBuffer buffer = ByteBuffer.allocate(dataBuffer.readableByteCount());
            dataBuffer.toByteBuffer(buffer);
            DataBufferUtils.release(dataBuffer);
            return buffer;
        }), this.parallelTransferOptions).block();
    }
```
And Here is the Example repo with solution https://github.com/git9999999/azure-blob-large-file-upload-problem

Here the Ticket that solve the Problem: https://github.com/Azure/azure-sdk-for-java/issues/35477

(Edit)

With below spring boot code, I can able to download very large file via REST and store it to an Azure Blobstorage.

Code:

import com.azure.storage.blob.BlobClient;
import com.azure.storage.blob.BlobClientBuilder;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

import java.io.ByteArrayInputStream;
import java.io.IOException;

@RestController
public class AzureBlobStorageServiceCon {

    @Value("<storage-account-connection-string>")
    private String connectionString;

    @Value("<container-name>")
    private String containerName;

    @GetMapping("/download")
    public ResponseEntity<String> uploadFile() throws IOException {

        RestTemplate restTemplate = new RestTemplate();
        byte[] fileBytes = restTemplate.getForObject("https://github.com/DasariKamali/blob/blob/main/10GB.zip", byte[].class);

        BlobClient blobClient = new BlobClientBuilder()
                .connectionString(connectionString)
                .containerName(containerName)
                .blobName("10GB.zip")
                .buildClient();

        ByteArrayInputStream inputStream = new ByteArrayInputStream(fileBytes);
        blobClient.upload(inputStream, fileBytes.length);

        return new ResponseEntity<>("File uploaded successfully", HttpStatus.OK);
    }
}

Output: