Question 38590 in Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Nginx server file storage – Ubuntu

I have such a task

I have file storage, about 50GB

Almost all files are 100KB -> 3MB.
And there are tracing files of ~25mb – but these files are downloaded very often

My task is to configure the software in such a way as to ensure maximum download performance from my Linux Server

Server:

  • System: Ubuntu 20
  • Disk: SSD NVMe
  • RAM 64 GB
  • CPU: 12
  • Internet: 1 Gbit/s

I tried the following combinations

  1. Nginx
  2. Varnish + Nginx

but I am facing the problem that the file download speed is up to 1 MB per second – even though I changed the settings in Nginx

  • Average download time 16 sec = 25MB – from my server
  • 2-3 seconds – 25MB – from the Firebase server

What software can be suitable for solving my problem? or in what direction do I need to look?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    For me the best solution turned out to be the solution https://github.com/minio/minio

    After installation and basic configuration, the installation on my server turned out to be quite productive Now I have ordered additional options for myself from the provider Such as 10G internet channel as well as additional ssd


  2. 0

    Varnish Cache, the open source version of Varnish, can handle up to 100 Gbps of throughput on a single machine. Since your NIC is limited to 1 Gbps.

    If you want the 50 GB catalog to be stored in the cache at all times, ensure you assign at least 50 GB to the varnishd runtime process by setting the -s option to the right value.

    Please keep in mind that Varnish also needs memory to process the workloads on the various threads.

    Make sure that the VCL configuration in /etc/varnish/default.vcl is properly configured to serve these files from the cache.

    If that files are static, you can assign a very long TTL to each cached object to ensure they are always served from the cache.

    If it turns out that the catalog of files grows and exceeds the 50 GB limit you set, either increase the size of the cache, or accept the fact that the least recently requested objects are going to be removed from the cache to free up space for new objects.

    Sizing your cache is a tradeoff and also depends on the frequency with which certain objects are fetched. The impact of a cache miss on your origin web server is also a factor to keep in mind.

    In general I would advise you to assign about 80% of your server’s memory to Varnish for caching. Unless you have crazy traffic spikes that go beyond the basic threading settings of Varnish, you’ll be fine.

    This is a very basic and high-level answer, but it validates Varnish as a candidate to accelerate the 50GB of content you’ve referred to in your question.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search