skip to Main Content

I’m running two Laravel apps in a high availability setup, with a load balancer directing traffic between them.

Both apps use the same Redis DB (AWS ElasticCache) for queues , and are set up with Laravel Horizon.
Both apps have the same configurations and 3 workers each. "high", "medium" and "low".
Most jobs are running fine, but there’s one job that takes longer than others and is causing an issue.
The failing job is running on the ‘low’ worker.

So the job is processed by one horizon. It’s processing and after 1 minute and 30 seconds, the second laravel horizon is also taking the job and start processing it. Since this job can’t run in parallel, the job fails.

It looks like the lock system isn’t working properly, since both Laravel Horizon instances are taking the job.

Does horizon have a lock system or do I have to implement my own ?

I also have no idea why 90s after the job is taken by horizon, the 2nd horizon is taking it.

config/horizon.php

'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['high', 'default', 'low'],
            'balance' => 'auto',
            'processes' => 1,
            'tries' => 1,
            'timeout' => 1250,
            'memory' => 2048,
        ],
    ],
],

config/queue.php

'connections' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default',
        'retry_after' => 1260,
        'block_for' => null,
    ],
],

2

Answers


  1. Chosen as BEST ANSWER

    This isn't super clear in the documentation but the solution was to specify a custom retry_after for all queues having longer process than 90s.

    To set that globally you can pass all the queues separated by a comma.

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'low,default,high',
        'retry_after' => 1260,
    ],
    

  2. WithoutOverlapping middleware should help you with that
    https://laravel.com/docs/10.x/queues#preventing-job-overlaps

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search