How to reference and upload a file in Azure Blob Storage?

Caverman
December 1, 2022
208 views
1 vote
2 Answers

I’m working on an Azure Function App that will grab a .pgp file off of Blob Storage, decrypt it, and then upload that decrypted file back to Blob Storage.

I’ve done quite a bit of research and everything usually assumes you are downloading a file to a local drive, decrypt, then upload. However, in my case I’m trying to do everything in Azure.

This is the code I’ve come up with so far. This will connect to and download the file to a stream successfully but I’m not figuring out how to wire it up with the output stream.
The line for the UploadAsync() is the one I’m having issues with and it needs a value passed into the method but I’m assuming the targetBlobClient already has reference to the Blob Container and file name.

I’m lost here and can’t seem to find any kind of examples to help me figure out what to do. I’m sure this code could be reduced and I will look into that once I can get it to work.

var outputStream = await targetBlobClient.UploadAsync();

Here is the code I’ve come up with so far:

try
            {

                var privateKeyValue = GetKeyVaultSecretValue(m.KeyVaultURL, m.KeyVaultPrivateSecretName);
                var privateKeyPassword = GetKeyVaultSecretValue(m.KeyVaultURL, m.KeyVaultPrivateSecretPassword);



                var storageConnString = m.InputStorageConnection;
                var containerName = m.InputStorageContainer;
                var sourceFile = m.InputFileName;
                var targetFile = m.OutputFileName;

                var sourceFolder = Path.Combine(m.InputStorageContainer, m.InputStorageFolder);
                var targetFolder = Path.Combine(m.OutputStorageContainer, m.OutputFolder);

                Console.WriteLine(@"Source full path: " + sourceFolder + "\" + sourceFile);


                BlobServiceClient blobServiceClient = new BlobServiceClient(storageConnString);

                BlobContainerClient sourceContainerClient = blobServiceClient.GetBlobContainerClient(sourceFolder);     
                BlobClient sourceBlobClient = sourceContainerClient.GetBlobClient(sourceFile);     

                BlobContainerClient targetContainerClient = blobServiceClient.GetBlobContainerClient(targetFolder);     
                BlobClient targetBlobClient = sourceContainerClient.GetBlobClient(targetFile);     

                if (await sourceBlobClient.ExistsAsync())
                { 
                    var inputStream = await sourceBlobClient.DownloadAsync();
                    var outputStream = await targetBlobClient.UploadAsync();

                    EncryptionKeys encryptionKeys = new EncryptionKeys(privateKeyValue, privateKeyPassword);

                    PGP pgp = new PGP(encryptionKeys);


                    await pgp.DecryptStreamAsync(inputStream, outputStream);

                }
                else
                {
                    Console.WriteLine(@"Error finding file. " + sourceFolder + "\" + sourceFile);
                    _log.LogError("Error find file {0}\{1}.", sourceFolder, sourceFile);
                }

            }
            catch (Exception ex)
            {
                _log.LogError("Error decrypting file. EventType: {0} | File: {1} | {2} | {3} | {4}", m.EventName, m.InputFileName, ex.Message, ex.StackTrace, ex.InnerException);

                Console.WriteLine("Error: " + ex.Message);
            }

Answers

Chosen as BEST ANSWER

I worked with someone and we finally figured it out. The code could probably be cleaned up some but this seems to work for now. This uses all memory streams to download, decrypt, upload. I'll eventually create an encryption side of the Function App as well. In PROD we will be using a Service Bus message to kick this off but I'm using an HTTP Trigger to make it easy to test with Postman. Hopefully it helps out someone else.

Also note that there is a known bug currently (Nov 2022) with adding Key Vault secret values through the Azure Portal. It strips out all of the formatting which renders the PGP Keys invalid. You have to use Azure Cloud Shell or similar to upload a multi-line file first and then use Powershell (or Bash) to insert the file value into Key Vault in order to keep the formatting.

public async Task DecryptFileAsync(PGPmessage m)
{
    _log.LogInformation("Start decryption process for Event: {0}", m.EventName);


    //=========  Get PGP Keys  ================================================================
    var privateKeyValue = GetKeyVaultSecretValue(m.KeyVaultURL, m.KeyVaultPrivateSecretName);
    var privateKeyPassword = GetKeyVaultSecretValue(m.KeyVaultURL, m.KeyVaultPrivateSecretPassword);


    try
    {
        var storageConnString = m.InputStorageConnection;

        var sourceFolder = m.InputStorageContainer;
        var sourceFile = m.InputFileName;
        
        var targetFolder = m.OutputStorageContainer;
        var targetFile = m.OutputFileName;

        _log.LogInformation(@"Looking for file {0}{1}.", sourceFolder, sourceFile);
        Console.WriteLine(@"Source full path: " + sourceFolder + "\" + sourceFile);


        // Create the connections to Blob Storage for both Source and Target files.
        BlobServiceClient blobServiceClient = new BlobServiceClient(storageConnString);

        BlobContainerClient sourceContainerClient = blobServiceClient.GetBlobContainerClient(sourceFolder);     
        BlobClient sourceBlobClient = sourceContainerClient.GetBlobClient(sourceFile);     

        BlobContainerClient targetContainerClient = blobServiceClient.GetBlobContainerClient(targetFolder);
        BlobClient targetBlobClient = sourceContainerClient.GetBlobClient(targetFile);

        if (await sourceBlobClient.ExistsAsync())
        {
            // Use a memory stream because a Stream type is not seekable, a memory stream is. 
            var inputStream = new MemoryStream();
            var outputStream = new MemoryStream();

            // Download from blob storage. Copy to memory stream so that it can be seakable. 
            // After copying to memory stream, reset the position to 0 so that it will be able to read from the begining.
            var blobDownloadStream = await sourceBlobClient.DownloadAsync();
            blobDownloadStream.Value.Content.CopyTo(inputStream);
            inputStream.Position = 0;

            // Create PGP Core object passing in the PGP Keys. 
            EncryptionKeys encryptionKeys = new EncryptionKeys(privateKeyValue, privateKeyPassword);
            PGP pgp = new PGP(encryptionKeys);

            await pgp.DecryptStreamAsync(inputStream, outputStream);

            _log.LogInformation(@"Uploading file to storage: {0}{1}", targetFolder, targetFile);

            // Reset to the beginning of the stream since it will be at the end due to writing the decrypted value to the stream.
            outputStream.Position = 0;          
            await targetBlobClient.UploadAsync(outputStream, true);     //Set to overwrite=true

            _log.LogInformation(@"Uploading to file to storage Complete. {0}{1}", targetFolder, targetFile);
            Console.WriteLine(@"Uploading to file to storage Complete. {0}{1}", targetFolder, targetFile);
        }
        else
        {
            Console.WriteLine(@"Error finding file: " + sourceFolder + "\" + sourceFile);
            _log.LogError("Error finding file: {0}\{1}.", sourceFolder, sourceFile);
        }
    }
    catch (Exception ex)
    {
        _log.LogError("Error decrypting file. EventType: {0} | File: {1} | {2} | {3} | {4}", m.EventName, m.InputFileName, ex.Message, ex.StackTrace, ex.InnerException);

        Console.WriteLine("Error: " + ex.Message);
    }
}

private string GetKeyVaultSecretValue(string keyVaultURL, string secretName)
{
    var kvSecretValue = string.Empty;

    try
    {
        var secretsClient = new SecretClient(new Uri(keyVaultURL), new DefaultAzureCredential());
        kvSecretValue = secretsClient.GetSecret(secretName).Value.Value;

        //https://scottgeek.technology/the-azure-vault-pgp-and-other-matters-part-2/
    }
    catch (Exception ex)
    {
        _log.LogError("Error getting Key Vault secret. SecretName: {0} | {1} | {2} | {3}", secretName, ex.Message, ex.StackTrace, ex.InnerException);
    }

    return kvSecretValue;
}

(Edit)

The UploadAsync method takes a stream. Here is the code from our Encryption routine that writes to Blob Storage:

// TMP file names
var temp_sourceFileName = IOHelper.BuildTempFileName(BlobHelper.StripPath(sourceBlobName));
var temp_targetFileName = IOHelper.BuildTempFileName(BlobHelper.StripPath(targetBlobName));
var temp_keyFileName = IOHelper.BuildTempFileName(BlobHelper.StripPath(keyBlobName));


// download Blob to TMP
using (var sourceStream = new FileStream(temp_sourceFileName, FileMode.Create))
{
    var sourceBlobClient = new BlobClient(blobAccountConnStr, sourceContainerName, sourceBlobName);
    await sourceBlobClient.DownloadToAsync(sourceStream);
}

// download key to TMP
using (var keyStream = new FileStream(temp_keyFileName, FileMode.Create))
{
    var keyBlobClient = new BlobClient(blobAccountConnStr, sourceContainerName, keyBlobName);
    await keyBlobClient.DownloadToAsync(keyStream);
}


// Encrypt stream
using (var pgp = new PGP())
{
    using (FileStream inputFileStream = new FileStream(temp_sourceFileName, FileMode.Open))
    {
        using (Stream outputFileStream = File.Create(temp_targetFileName))
        {
            using (Stream publicKeyStream = new FileStream(temp_keyFileName, FileMode.Open))
            {
                pgp.EncryptStream(inputFileStream, outputFileStream, publicKeyStream, true, true);
            }
        }
    }
}

// write to target blob
// write to target blob
using (var encryptStream = new FileStream(temp_targetFileName, FileMode.Open))
{
    var targetBlobClient = new BlobClient(blobAccountConnStr, targetContainerName, targetBlobName);
    await targetBlobClient.UploadAsync(encryptStream, true);

    return new OkObjectResult(targetBlobClient);
}

Please signup or login to give your own answer.

Click here to cancel reply.