Shopify - Laravel 11: job rate limit to not exceed rate limit of external API

Jan
August 25, 2024
206 views
0 votes
2 Answers

since Laravel 11 queues can also have a rate limit now (https://laravel.com/docs/11.x/queues#rate-limiting). My Laravel application is doing some requests to the Shopify API to fetch new products, add notes to some orders and also adding shipment information such as a tracking number to the order.

I have a Shopify Basic plan which is allowed to make a max of 2 requests per second but no more than 40 requests per minute.

Now I want to write a universal job which I can utilize to make calls to the Shopify API. So either I am fetching products or updating products, I want to have one single job class which takes care of that, so I can make sure I am not exceeding the Shopify API rate limit.

However, I am not able to make this work. I have defined a rate limit in my AppServiceProvider.php as described in the Laravel documentation:

public function boot(): void
{
    // Rate limit Shopify API requests set to 2 per second and 40 per minute
    RateLimiter::for('shopify-api-requests', function (object $job) {
        return [
            Limit::perSecond(2),
            Limit::perMinute(40),
        ];
    });
}

This is my reusable job class, which I want to reuse for every request I make to the Shopify API:

class ShopifyApiRequestJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $endpoint;
    public $method;
    public $data;
    
    public function __construct(string $endpoint, string $method = 'GET', array $data = null)
    {
        $this->endpoint = $endpoint;
        $this->method = $method;
        $this->data = $data;
    }
    
    public function backoff(): array
    {
        return [1, 5, 10];
    }
    
    public function tries(): int
    {
        return 3;
    }
    
    public function middleware(): array
    {
        return [
            new RateLimited('shopify-api-requests'),
            //new WithoutOverlapping('shopify-api-requests')
        ];
    }
    
    public function handle()
    {
        // Construct the full URL
        $url = 'https://' . config('settings.SHOPIFY_API_DOMAIN') . '/admin/api/' . config('settings.SHOPIFY_API_VERSION') . '/' . $this->endpoint;

        $response = Http::withHeaders([
            'X-Shopify-Access-Token' => config('settings.SHOPIFY_API_KEY'),
            'Content-Type' => 'application/json',
        ])->{$this->method}($url, $this->data);

        // Handle the response as needed (e.g., log it, store it, etc.)
        if ($response->failed()) {
            // Handle failure (e.g., retry the job, log the error, etc.)
            Log::error("Shopify API Request Failed (" . $response->status() . "): " . $response->body() . " " . $url);
        } else {
            // Handle success (e.g., process the response, store it, etc.)
            Log::info("Shopify API Request Successful");
        }
    }
}

When I test my job class, it does not behave as expected. I have created a foreach loop and have dispatched my job 10 times.

The expected result I am trying to archive is that every job which got dispatched is not overlapping with another job of that same class (ShopifyApiRequestJob) and per second are only 2 jobs being processed max and per minute 30 jobs max.

However, I end up with a log like this:

[2024-08-24 19:29:11] local.INFO: Shopify API Request Successful  
[2024-08-24 19:29:17] local.INFO: Shopify API Request Successful  
[2024-08-24 19:29:20] local.INFO: Shopify API Request Successful  
[2024-08-24 19:29:26] local.ERROR: AppJobsShopifyApiRequestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): [...]
[2024-08-24 19:29:26] local.ERROR: AppJobsShopifyApiRequestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): [...]
[2024-08-24 19:29:26] local.ERROR: AppJobsShopifyApiRequestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): [...]
[2024-08-24 19:29:26] local.ERROR: AppJobsShopifyApiRequestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): [...]
[2024-08-24 19:29:26] local.ERROR: AppJobsShopifyApiRequestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): [...]
[2024-08-24 19:29:26] local.ERROR: AppJobsShopifyApiRequestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): [...]
[2024-08-24 19:29:26] local.ERROR: AppJobsShopifyApiRequestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): [...]

Three jobs are being processed sucessfully but all other 7 jobs fail because of MaxAttemptsExceededException. I have increased the $backOff time on purpose to debug it but was not successful.

I don’t understand what I did wrong, configuring my job. I have followed the documentation.
Anybody can give me an advice on how to solve this problem?

Furthermore, I would like to receive a notification if all retries of a job have failed and not for every retry.

Anybody knows how to archive this behavior?

Kind regards

Answers

To effectively handle Shopify API rate limits using Laravel’s queue system, you need to address a few key points in your implementation.

Please follow these steps and hope it is solved:

First Rate Limiting Configuration:

AppServiceProvider Configuration

use IlluminateSupportFacadesRateLimiter;

public function boot(): void
{
    RateLimiter::for('shopify-api-requests', function () {
        return [
            Limit::perSecond(2)->by('shopify-api'),
            Limit::perMinute(40)->by('shopify-api'),
        ];
    });
}

Second Job Configuration

Job Class

use IlluminateBusQueueable;
use IlluminateContractsQueueShouldQueue;
use IlluminateFoundationBusDispatchable;
use IlluminateQueueInteractsWithQueue;
use IlluminateQueueSerializesModels;
use IlluminateSupportFacadesHttp;
use IlluminateSupportFacadesLog;
use IlluminateRedisLimit;

class ShopifyApiRequestJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $endpoint;
    public $method;
    public $data;

    public function __construct(string $endpoint, string $method = 'GET', array $data = null)
    {
        $this->endpoint = $endpoint;
        $this->method = $method;
        $this->data = $data;
    }

    public function backoff(): array
    {
        return [2, 10, 20]; // Adjusted backoff times
    }

    public function tries(): int
    {
        return 5; // Increase tries to accommodate retries within the rate limits
    }

    public function middleware(): array
    {
        return [
            new RateLimited('shopify-api-requests'),
        ];
    }

    public function handle()
    {
        // Construct the full URL
        $url = 'https://' . config('settings.SHOPIFY_API_DOMAIN') . '/admin/api/' . config('settings.SHOPIFY_API_VERSION') . '/' . $this->endpoint;

        $response = Http::withHeaders([
            'X-Shopify-Access-Token' => config('settings.SHOPIFY_API_KEY'),
            'Content-Type' => 'application/json',
        ])->{$this->method}($url, $this->data);

        // Handle the response
        if ($response->failed()) {
            // Log failure
            Log::error("Shopify API Request Failed (" . $response->status() . "): " . $response->body() . " " . $url);

            // If all retries fail, log a separate error
            if ($this->attempts() >= $this->tries()) {
                Log::critical("All retries for Shopify API Request Failed: " . $url);
            }

            // Re-throw the exception to allow Laravel to handle retries
            throw new Exception("Shopify API request failed.");
        } else {
            // Log success
            Log::info("Shopify API Request Successful");
        }
    }
}

Third Handling Retries and Notifications

Job Failure Handling

use IlluminateSupportFacadesNotification;
use AppNotificationsJobFailedNotification;

public function failed(Exception $exception)
{
    // Notify the admin when all retries have failed
    Notification::route('mail', config('settings.ADMIN_EMAIL'))
        ->notify(new JobFailedNotification($this->endpoint, $exception));
}

I’ve faced similar issue related to Rate Limit with external API.

Can you remove tries method and add retryUntil method

public function retryUntil(): DateTime
{
    return now()->addMinutes(IlluminateSupportCarbon::MINUTES_PER_HOUR * 2);
}

Here is the My Job implementation

<?php

namespace AppJobsReportPrtgReport;

use AppModelsUser;
use DateTime;
use IlluminateBusBatchable;
use IlluminateBusQueueable;
use IlluminateContractsQueueShouldQueue;
use IlluminateFoundationBusDispatchable;
use IlluminateQueueInteractsWithQueue;
use IlluminateQueueMiddlewareWithoutOverlapping;
use IlluminateQueueSerializesModels;
use IlluminateSupportCarbon;
use IlluminateSupportFacadesCache;
use IlluminateSupportFacadesHttp;

class SendExternalRequest implements ShouldQueue
{
    use Batchable;
    use Dispatchable;
    use InteractsWithQueue;
    use Queueable;
    use SerializesModels;

    /**
     * The number of seconds the job can run before timing out.
     *
     * @var int
     */
    public $timeout = 120;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct(User $user)
    {
        //
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $response = Http::baseUrl('mybaseurl.com')
            ->get('test.php', []);

        if ($response->successful()) {
            //logic Goes here

        } else {
            $this->release(10);
        }
    }

    /**
     * Get the middleware the job should pass through.
     *
     * @return array<int, object>
     */
    public function middleware(): array
    {
        return [
            new APIRateLimiterMiddleware(),

            (new WithoutOverlapping(User::class.':HIT:'.$this->user->id))
                ->releaseAfter(10),
        ];
    }

    /**
     * Determine the time at which the job should timeout.
     */
    public function retryUntil(): DateTime
    {
        return now()->addMinutes(Carbon::MINUTES_PER_HOUR * 2);
    }
}

And Here is the middleware that I use

<?php

namespace AppJobsMiddleware;

use AppModelsUser;
use Closure;
use IlluminateSupportFacadesRedis;

class APIRateLimiterMiddleware
{
    /**
     * Process the queued job.
     *
     * @param  Closure(object): void  $next
     */
    public function handle(object $job, Closure $next): void
    {
        Redis::throttle('API'.':Screenshot')
            ->block(0)
            ->allow(1)
            ->every(3) // Allow 1 job every 3 seconds
            ->then(function () use ($job, $next) {
                // Lock obtained...
                $next($job);
            }, function () use ($job) {
                // Could not obtain lock...
                $job->release(3); // Release the job back to the queue after 3 seconds
            });
    }
}

The code I’ve Provided is just a sample from my implementation pls
make sure to adjust to your requirements.

Please signup or login to give your own answer.

Click here to cancel reply.

Shopify – Laravel 11: job rate limit to not exceed rate limit of external API

Answers