One of the features of the application which I am currently working on is photo upload. Customers upload photos in frontend and photos are passed to rails backed and then stored on Amazon S3.
I have noticed that a huge amount of request time is spent uploading photos to s3. The photos are uploaded one by one so latency is multiplied. It would be great if I could somehow store photos temporarily in RAM and increase request speed.
I have thinked about running a Sidekiq job with a file as params but according to sidekiq documentation passing a huge object is not good practise. How can I solve this in another vay ?
2
Answers
I think this problem by using an API to generate a presigned url and using cognito to upload the image on s3 and get the image link.
nginx/puma running on machine A should save the image as a local file. Run Sidekiq on the same machine A and pass the filename to a job in a host-specific queue for Sidekiq to process. That way you can pass a file reference without worrying which machine will process it.
Make sure Sidekiq deletes the file so you don’t fill up the disk!
https://www.mikeperham.com/2013/11/13/advanced-sidekiq-host-specific-queues/