Json - Uploading a large file issues

Freshman39sDream
December 5, 2022
276 views
1 vote
2 Answers

I am using Blazors InputFile component and I wish to upload and parse Json from a web view. Upon uploading a very large json file (11mb and ~300.000 lines) I got an exception that the json content is invalid. I tried putting a breakpoint on the line above, and it seems to be sliced somehow, so the end of the file is in the middle of an element. I checked the stream (see the code below) and after some index (around 100.000), every element has value of 0.

razor

<InputFile @ref="InputFile" OnChange="OnUploadHandler" />

razor.cs

    async Task OnUploadHandler(InputFileChangeEventArgs e)
    {
        var stream = e.File.OpenReadStream(Int32.MaxValue);
        Int32 length = stream.Length > Int32.MaxValue ? Int32.MaxValue : Convert.ToInt32(stream.Length);
        Byte[] buffer = new Byte[length];

        await stream.ReadAsync(buffer.AsMemory(0, length));
        var content = Encoding.Default.GetString(buffer);
        var parsed = JObject.Parse(content);
    }

Consider a toy example. Suppose I upload a file lorem.json that has the following content:

[
  {
    "_id": "638dfa2f5e1dd23eb7b407e9",
    "index": 0,
    "guid": "89e63cb9-0498-4f5f-829e-9aaec0cf6855",
    "address": "706 Irving Avenue, Leroy, Iowa, 6592",
    "about": "Cupidatat adipisicing cupidatat culpa et labore id proident incididunt occaecat. Laboris velit fugiat consequat est sunt veniam ex. Ullamco irure ex laborum dolor esse anim et sunt in nulla ea aliquip. Ea proident irure cillum quis qui magna id ad ex qui. Veniam anim Lorem aliqua officia. Et cillum dolore ad aliqua ex voluptate culpa dolor occaecat cupidatat ex qui et enim. Elit irure voluptate non cupidatat aliqua duis in duis ut qui.rn",
    "tags": [
      "irure",
      "tempor",
      "irure",
      "consectetur",
      "duis",
      "proident",
      "ex"
    ],
    "friends": [
      {
        "id": 0,
        "name": "Zimmerman Burch"
      },
      {
        "id": 1,
        "name": "Snow Harrington"
      }
    ],
    "greeting": "Hello, undefined! You have 3 unread messages.",
    "favoriteFruit": "strawberry"
  },
  {
    "_id": "638dfa2f226b8c0be4f73fa9",
    "index": 1,
    "guid": "0a7b9bb4-580f-460e-b876-f0cf62260983",
    "address": "241 Jamaica Avenue, Chicopee, Palau, 891",
    "about": "Excepteur laboris reprehenderit consequat labore exercitation enim consequat. Ea cillum in nulla deserunt amet ipsum ad magna Lorem minim sunt veniam magna. Laboris voluptate aute ullamco labore dolor consectetur irure aute ea cillum aliquip esse dolor ex. Labore proident officia nostrud excepteur eu ad irure ipsum sint cupidatat eiusmod esse. Exercitation ea incididunt consequat laborum cillum incididunt reprehenderit exercitation enim nostrud elit dolor pariatur ex. Magna cillum Lorem deserunt minim ad irure do sit non velit qui culpa veniam occaecat. Sunt culpa non laborum Lorem nisi ea laborum proident occaecat pariatur aliquip irure eu.rn",
    "tags": [
      "aute",
      "labore",
      "consequat",
      "do",
      "est",
      "labore",
      "ex"
    ],
    "friends": [
      {
        "id": 0,
        "name": "Johanna Dunn"
      },
      {
        "id": 1,
        "name": "Prince Estes"
      }
    ],
    "greeting": "Hello, undefined! You have 7 unread messages.",
    "favoriteFruit": "strawberry"
  }
]

in the code above, if I were to put a breakpoint and look at the content variable, I would find it to be something like:

[
  {
    "_id": "638dfa2f5e1dd23eb7b407e9",
    "index": 0,
    "guid": "89e63cb9-0498-4f5f-829e-9aaec0cf6855",
    "address": "706 Irving Avenue, Leroy, Iowa, 6592",
    "about": "Cupidatat adipisicing cupidatat culpa et labore id proident incididunt occaecat. Laboris velit fugiat consequat est sunt veniam ex. Ullamco irure ex laborum dolor esse anim et sunt in nulla ea aliquip. Ea proident irure cillum quis qui magna id ad ex qui. Veniam anim Lorem aliqua officia. Et cillum dolore ad aliqua ex voluptate culpa dolor occaecat cupidatat ex qui et enim. Elit irure voluptate non cupidatat aliqua duis in duis ut qui.rn",
    "tags": [
      "irure",
      "tempor",
      "irure",
      "consectetur",
      "duis",
      "proident",
      "ex"
    ],
    "friends": [
      {
        "id": 0,
        "name": "Zimmerman Burch"
      },
      {
        "id": 1,
        "name": "Snow Harrington"
      }
    ],
    "greeting": "Hello, undefined! You have 3 unread messages.",
    "favoriteFruit": "strawberry"
  },
  {
    "_id": "638dfa2f226b8c0be4f73fa9",
    "index": 1,
    "guid": "0a7b9bb4-580f-460e-b876-f0cf62260983",

Tags: .net blazor c#file-upload json

Answers

Hello you need to specify maxallowedSize

This is my method for handling this:

    public async Task AddImage(IBrowserFile selectedFile)
    {
        long maxallowedsize = 1024*3000;
        var anonymizedFileName = $"{Guid.NewGuid().ToString()}{selectedFile.Name.Substring(selectedFile.Name.IndexOf('.'))}";
        var path = Path.Combine(_environment!.ContentRootPath, "wwwroot/gallery", anonymizedFileName);
        await using FileStream fs = new(path, FileMode.Create);
        await selectedFile.OpenReadStream(maxallowedsize).CopyToAsync(fs);
        using var factory = _contextFactory.CreateDbContext();
        await factory.DarknetGallery.AddAsync(new DarknetGalleryEntity
        {
            ImageFileName = anonymizedFileName,
            CreationTime = DateTime.Now,
            IsVerified = false
        });
        await factory.SaveChangesAsync();
    }

but the thing is here:
(1024*3000) = this is max size of uploaded file, you need to calculate this first.

 long maxallowedsize = 1024*3000;
 await selectedFile.OpenReadStream(maxallowedsize).CopyToAsync(fs);

How to do it without loading a huge file into memory. This is only the relevant parts of my code. Hope there is enough to get you started. I limited this App to 30mb. (31457280 bytes)

Note: The FileService and ApiBroker are injected via interface for testing.

Uploader.razor.cs

    private async Task OnInputFileChange(InputFileChangeEventArgs inputFileChangeEventArgs)
    {
        const long maxFileSize = 31457280;
        var browserFiles = inputFileChangeEventArgs.GetMultipleFiles(maxAllowedFiles);
        await FileService.PostFilesAsync(maxFileSize, browserFiles);
    }

FileService.cs

    public async ValueTask PostFilesAsync(long maxFileSize, IReadOnlyList<IBrowserFile> browserFiles)
    {
        using FileUploadContent fileUploadContent = new FileUploadContent();

        MultipartFormDataContent dataContent = new MultipartFormDataContent();

        fileUploadContent.Upload = AddSelectectFiles(maxFileSize, fileUploadContent, browserFiles);

        if (fileUploadContent.Upload)
        {
            HttpResponseMessage response = await apiBroker.PostFilesAsync(fileUploadContent.MultipartFormDataContent);

            IList<UploadResult>? newUploadResults = await response.Content.ReadFromJsonAsync<IList<UploadResult>>();

            if (newUploadResults is not null)
            {
                uploadResults = uploadResults.Concat(newUploadResults).ToList();
            }
        }
    }

    private bool AddSelectectFiles(long maxFileSize, FileUploadContent fileUploadContent, IReadOnlyList<IBrowserFile> browserFiles)
    {
        bool upload = false;

        foreach (var browserFile in browserFiles)
        {
            if (uploadResults.SingleOrDefault(f => f.FileName == browserFile.Name) is null)
            {
                try
                {
                    AddFileToFormContent(maxFileSize, fileUploadContent, browserFile);

                    upload = true;
                }
                catch (Exception ex)
                {
...
                }
            }
        }

        return upload;
    }

    private void AddFileToFormContent(long maxFileSize, FileUploadContent fileUploadContent, IBrowserFile browserFile)
    {
        System.IO.Stream fileStream = browserFile.OpenReadStream(maxFileSize);
        StreamContent fileContent = new StreamContent(fileStream);
        fileContent.Headers.ContentType = new MediaTypeHeaderValue(browserFile.ContentType);

        File file = new File()
        {
            Name = browserFile.Name,
            Length = browserFile.Size,
            ContentType = browserFile.ContentType,
            UploadStream = fileStream
        };

        this.files.Add(item: file);

        fileUploadContent.MultipartFormDataContent.Add(content: fileContent, name: ""files"", fileName: file.Name);
    }

ApiBroker.cs

    public async ValueTask<HttpResponseMessage> PostFilesAsync(MultipartFormDataContent multipartFormDataContent)
    {
        return await httpClient.PostAsync(requestUri: "api/UserFiles", multipartFormDataContent);
    }

Please signup or login to give your own answer.

Click here to cancel reply.

Json – Uploading a large file issues

Answers