skip to Main Content

There are a lot of places on the internet that show how to return files but I have found none that will return dynamically generated binary data without storing the whole contents on memory. Maybe I should serialize my data using Json instead of protobufers.

Thanks to this question I was able to create something like this:

[HttpGet]
public ActionResult DownloadItems()
{
    // get 100K items from database as IEnumerable.
    IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    // create memory stream where to place serialized items
    MemoryStream ms = new ();

    // write all serialized items to stream
    foreach(var item in items)
    {
         byte[] itemSerialized = item.BinarySerialize();
         ms.Write(itemSerialized,0,itemSerialized.Length);
    }

    // set position to the begining of memory stream
    ms.Position = 0;

    return File(ms, "application /octet-stream", "foo.bin");
}

This works well but I am loading 100K items into memory. My question is how can I return the same dynamically generated file without having to load all the items into memory?

I remember that the HTTP protocol returns something like this when returning binary files:


HTTP response headers
...

---------SomeGUID--------------

.. binary data

---------SomeGUID--------------

as a result I believe that having something like this will make it work (it has pseudo code):

[HttpGet]
public ActionResult DownloadItems()
{
    // get 100K items from database as IEnumerable.
    IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    // write the begining of file (PSEUDO code)
    this.response.body.writeString("-----------------SomeGuid------------");

    // write all serialized items to stream
    foreach(var item in items)
    {
         byte[] itemSerialized = item.BinarySerialize();
         this.response.body.write(itemSerialized,0,itemSerialized.Length);
    }

    // set position to the begining of memory stream
    ms.Position = 0;

    this.response.body.writeString("-----------------SomeGuid------------");
}

I can install fiddler or any other proxy to see how the real binary transfer of a file looks like. But is there a build in way of doing that so I don’t have to go through all that trouble?

2

Answers


  1. Chosen as BEST ANSWER

    I just created my own fake file stream for this to work:

    public class FakeFileStream : Stream
    {
        private readonly IEnumerator<object> _enumerator;
        private bool _completed;
    
        public FakeFileStream(IEnumerable<object> items)
        {
            if (items is null)
                throw new ArgumentNullException();
    
            _enumerator = items.GetEnumerator();
        }
    
    
        public override int Read(byte[] buffer, int offset, int count)
        {
            if (_enumerator.MoveNext())
            {
                var currentItem = _enumerator.Current;
    
                // deserialize item.  
                byte[] itemSerialized = currentItem.SerializeUsingDotNetProtoBuf();
    
                // this will probably not happen but it is a good idea to have it implemented.
                // if this is the case store data on memory and return it on the next read
                if (itemSerialized.Length > buffer.Length)
                    throw new NotImplementedException();
    
                // copy data to buffer
                Buffer.BlockCopy(itemSerialized, 0, buffer, 0, itemSerialized.Length);
                return itemSerialized.Length;
            }
            else
            {
                _completed = true;
                return 0;
            }
        }
    
        // unused methods
        public override void Flush() => throw new Exception("Unused method");
        public override long Seek(long offset, SeekOrigin origin) => throw new Exception("Unused method");
        public override void SetLength(long value) => throw new Exception("Unused method");
        public override void Write(byte[] buffer, int offset, int count) => throw new Exception("Unused method");
    
        // Properties
        public override bool CanRead => !_completed;
        public override bool CanSeek => false;
        public override bool CanWrite => false;
        public override long Length => throw new NotImplementedException("Not needed");
        public override long Position
        {
            get => throw new Exception("Unused property");
            set => throw new Exception("Unused property");
        }
    
        // Implement IDisposable
        public override ValueTask DisposeAsync()
        {
            _enumerator.Dispose();
            return base.DisposeAsync();
        }
    }
    

    and My endpoint looks like this:

    [HttpGet]
    [ProducesResponseType(StatusCodes.Status200OK, Type = typeof(FileStreamResult))]
    public IActionResult GetBinary()
    {         
        // get some IEnumerable collection 
        IEnumerable<Foo> items = MyDatabase.MyTable.Find("my query");
    
        // create a fake file stream
        var fs = new FakeFileStream(items);
    
        return File(fs, "application/octet-stream"); 
    }
    

    Edit

    Do not use sugested FakeFileStream. For some reason it gave me problems. Maybe because it was writting to the stream very few bytes.

    Anywyas I was not able to do this from a controller. But I was able to do it using middleware. I had to do simething like this:

    var builder = WebApplication.CreateBuilder(args);
    var app = builder.Build();
    
    app.UseHttpsRedirection();
    
    // any previous middleware you may have
    
    app.Use(async (context, next) =>
    {
        var downloadFile = context.Request.Query["downloadFile"];
        if (!string.IsNullOrWhiteSpace(downloadFile))
        {
            context.Response.ContentType = "application/octet-stream";
    
            // get 100K items from database as IEnumerable.
            IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");
    
        
    
            // write all serialized items to stream
            foreach(var item in items)
            {
                 byte[] itemSerialized = item.BinarySerialize();
                 await context.WriteAsync(itemSerialized, context.CancelationToken);
            }
    
            return;
    
      
        }
    
    
        // Call the next delegate/middleware in the pipeline.
        await next(context);
    });
    
    // etc rest of your middleware
    
    app.Run();
    

  2. Rather than trying to reuse File() / FileStreamResult, I would recommend implementing your own ActionResult and rendering the content to the response stream there.

    public class ByteStreamResult : ActionResult
    {
        private readonly IEnumerable<byte[]> blobs;
    
        public ByteStreamResult(IEnumerable<byte[]> blobs)
        {
            this.blobs = blobs;
        }
    
        public override async Task ExecuteResultAsync(ActionContext context)
        {
            context.HttpContext.Response.ContentType = "application/octet-stream";
            foreach (var item in blobs)
                await context.HttpContext.Response.Body.WriteAsync(item, context.HttpContext.RequestAborted);
        }
    }
    
    return new ByteStreamResult(items.Select(i => i.BinarySerialize()));
    

    Or you could go one step further and implement a custom formatter.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search