Question posted in 
I currently have an api project that is kind of a middleware API. It get’s requests and redirects it to other APIs and caches some things in redis for faster access.
Therefore I don’t really need a database and didn’t set any up.
As it also sends out e-mails I would like to set-up the queue to use redis. As far as I understand this is quite simple changing the queue driver to redis.
What I didn’t figure out is if I need a database for the failed jobs (as Laravel is usually creating that out of the box for that).
My further plan is to use horizon as a dashboard for my queues. As far as I understand horizon also saves the failed jobs. I found a comment that they are being flushed from time to time which was written seven years ago. So I am not sure if something changed there.
So my question is: Do I need to setup a database (e.g. mysql) to make sure no failed_jobs get lost or can I solve this by using horizon?
The FailedJobsController in horizon is using the JobRepository which is implemented by the RedisJobRepository
So far I didn’t setup the whole project because I didn’t know how to figure out if horizon really flushes the failed jobs.
2
Answers
If you are using Redis as your queue driver you don’t need a database like MySQL to handle the jobs.
How about failed jobs, though? Here is a brief explanation of how it works:
Queue:
Upon switching to Redis as your queue driver, Laravel will process your jobs and manage them in accordance with your queue configurations in config/queue.php. If the particular jobs fail, Laravel will do its best based on its settings in the config to work on the jobs again.
Failed Jobs
The support for failed jobs in the case of Redis is also present. Laravel by default uses a database table to keep the log data on failed jobs. Nevertheless, you don’t have to have an SQL database to deal with failed jobs in the case of Redis.
Laravel Horizon
Failed jobs storage: Horizon keeps records of each job in Redis also on the basis of a dashboard and a nice interface. The real jobs that did not stack properly can be reviewed in the dashboard for Horizon as well. The failed jobs will be stored in Redis as long as Horizon is correctly setup and running.
What about their persistence in the Redis database? If the configuration, by the way, Horizon works does not automatically clear the failed jobs unless you want them to. This information can be changed – you can define for how many hours the Redis should gather the failed jobs data and act in the Laravel configuration.
No database is needed
Since Horizon is your best choice here and it manages the failed jobs with Redis, you do not need a MySQL database just for your failed jobs project. The package Horizon already manages the failed jobs via its RedisJobRepository so they are not otherwise being erased unless the configuration says so or you do it. The Horizon configuration means you can do more on how and when data is erased from Redis by default.
Check configuration and logs: Be sure to check out Horizon settings as well as Redis configuration. Ensure that your Redis can handle the persistence and give you security of your jobs and failed jobs as needed. Also consider setting up logging so that you know what happens when these jobs fail to store themselves correctly.
Yes, by default failed jobs older than 7 days are cleared by horizon. The implementation is found here. The listener is triggered on every loop of the master worker. You can increase this limit by setting
horizon.trim.failedin the config.That said, it is there to avoid running out of memory, save yourself some headache and configure a database. SQLite is straightforward enough to set up.