I am trying to upload a parquet file to Azure, but it keeps giving me an error message saying that its missing a required HTTP header.
In the example I am providing I just tried some basic data in a memory stream and it is still throwing this exception.
RequestFailedException: An HTTP header that’s mandatory for this
request is not specified.
RequestId:0bd87df1-701f-0003-5ff4-944ccd000000
Time:2024-04-22T20:36:16.2840714Z Status: 400 (An HTTP header that’s
mandatory for this request is not specified.) ErrorCode:
MissingRequiredHeaderContent: {"error":{"code":"MissingRequiredHeader","message":"An HTTP
header that’s mandatory for this request is not
specified.nRequestId:…nTime:2024-04-22T20:36:16.2840714Z"}}Headers: Connection: keep-alive x-ms-error-code: MissingRequiredHeader
x-ms-request-id: … x-ms-version:
2024-05-04 x-ms-client-request-id:
… Content-Length: 204 Content-Type:
application/json;charset=utf-8 Date: Mon, 22 Apr 2024 20:36:16 GMT
Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0 Via: REDACTED
Code:
string storageAccountName = "name";
string storageAccountKey = "key";
StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(storageAccountName, storageAccountKey);
BlobServiceClient blobServiceClient = new BlobServiceClient(
new Uri("https://" + storageAccountName + ".dfs.core.windows.net"),
sharedKeyCredential
);
BlobContainerClient desContainer = blobServiceClient.GetBlobContainerClient("testcontainer");
BlobClient desBlob = desContainer.GetBlobClient("/testAPI/my.parquet");
var myStr = "Hello!";
var content = Encoding.UTF8.GetBytes(myStr);
using (var ms = new MemoryStream(content))
desBlob.Upload(ms);
I am not able to find what headers are missing and I am not sure how to add them as there does not seem a standard way to add headers besides; Content-Type
Packages:
- Azure.Core – 1.39.0
- Azure.Storage.Blobs – 12.20.0-beta.2
- Azure.Storage.Common – 12.19.0-beta.2
I’ve tried reverting to older packages thinking maybe functionality was there before but changed in recent revisions.
Any help in the right direction would be helpful.
2
Answers
As mentioned in the comments, please change the endpoint suffix from
dfs.core.windows.net
toblob.core.windows.net
. My guess is that when the REST API seesdfs.core.windows.net
endpoint, it tries to use Data Lake REST API instead of Blob Storage REST API and because of that, you are getting the missing header error.Parquet files are binary, so you shouldn’t use UTF8 encoding.
Remove UTF encoding from your code: