Redis – Laravel dispatch a Job after finishing the Job racks up memory and then fails

What I would like to do is dispatch a Job, and then continuing to dispatch the same Job after the previous one was finished so there is a continuous loop of the Job being dispatched. As indicated under option one this was previously working with the database queue driver.

I use Redis now so I can more easily monitor my jobs via Horizon.

To start the first job I use a custom artisan command, but this could as well be from within a controller, it’s just the first dispatch.

My config/horizon (local config is the same as production):

'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['arbitrage'],
                'balance' => 'auto',
                'processes' => 2,
                'tries' => 1,
            ],
            'supervisor-2' => [
                'connection' => 'redis',
                'queue' => ['trade'],
                'balance' => 'auto',
                'processes' => 4,
                'tries' => 1,
            ],
            'supervisor-3' => [
                'connection' => 'redis',
                'queue' => ['balance', 'trade_meta'],
                'balance' => 'auto',
                'processes' => 5,
                'tries' => 1,
            ],
            'supervisor-4' => [
                'connection' => 'redis',
                'queue' => ['notifications'],
                'balance' => 'auto',
                'processes' => 2,
                'tries' => 1,
            ],
        ],

Option 1: dispatch a new job at the end of the job

At the end of this Job’s handle() I re-dispatch it so that it runs continuously.

This actually worked fine (the process ran for weeks on end) when using the database driver.

Job:

class ArbitrageJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $tries = 1;
    public $timeout = 30;

    // Process things here

    ArbitrageJob::dispatch()->onQueue('arbitrage');
}

Option 2: dispatch a new job with Queue:after

Job:

class ArbitrageJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $tries = 1;
    public $timeout = 30;
}

Then in AppServiceProvider:

Queue::after(function (JobProcessed $event) {
        ProcessArbitrage::dispatch()->onQueue('arbitrage');
});

What I see happening now, for both options, is that for every time it runs, it’s racking up memory (I log this via memory_get_usage(true)).

The first run it’s 28MB, the next one it’s 32MB, and it keeps increasing until it’s around 122MB and then I get the following error: IlluminateQueueMaxAttemptsExceededException: AppJobsArbitrageJob has been attempted too many times or run too long. The job may have previously timed out.

Probably related: in config/horizon I’ve set memory_limit to 768MB but it looks like it fails a lot earlier than that. Or is that because it limits the memory to 128MB per worker?