php Question posted in
Official manual can be found here.

Running a long script in PHP

I’m trying to upload over 2,000 products with images from XML to my site. I wrote a script that does this. But this script takes a very long time and I can’t get the upload status of each product.

I don’t want advice like "make the execution time of the PHP script longer". I want to understand how to run very big scripts with PHP, divide tasks, and monitor each product’s loading status.

Tags:

2

Answers


  1. 0

    In such cases pagination is the answer.

    So you set limit and offset on how many you process each iteration.

    So you have a CRON JOB, ajax or any other way visit url with page parameter.

    /myscript.php?page=1

    then, in script.php would need to have:

    $limit = 250; // i.e. how many entries/rows/products to get.
$offset = ($limit * ( $_GET['page] - 1 ));

    your database query would need to have this limit and offest taken into account and then you do your processing of these products.

    That is a "summary" of the approach.

    You might want to have a controller script, that would read your XML file first, once a day, to find out how many entries there will be dealt with, then create a database that have a "queue" of work that needs to be done, pagination and statuses.. i.e. "not processed", "completed" and timestamps of when job was created and processed.

    Then outside of this controller, have a cron job that looks up to database of table at certain intervals to process this products in chunks.

    Login or Signup to reply.
  2. 0

    I would definitely use some sort of queueing system whether it’s rabbit, sqs, etc. This way you can parse records as they come into the system and offload tasks to individual queues. One queue would be responsible for handling database inserts, one for processing images, etc.

    This would provide some easy ways to monitor each part of the process and provide context when and if things go wrong.

    Here’s a really quick howto for PHP and rabbitMq: https://www.rabbitmq.com/tutorials/tutorial-one-php.html

    Cheers!

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search