Question posted in
Some documentation to get you started can be found here.

Prestashop background bulk product import – Apache

I’m trying to import efficiently 15K+ products periodically in a Prestashop e-commerce parsing an ASCII file (3.5MB) and using only Prestashop’s API.

All run under docker, with official images from docker hub.

If a product with the same reference field does not exists I’ll have to insert a new product, if it’s present to update it.
I developed a module that makes this via a click un a custom admin tab, and it works, but the whole system freezes till the process is finished or terminated: (almost 77 minutes).
I also tried to split the (not so) big file in chunks of 500, 100, 50 but the time for the processing decreases linearly, it doesn’t help very much:

  • 153 avg seconds for 500 elements
  • 31 avg seconds for 100 elements
  • 15 avg seconds for 50 elements

I could surely configure a cron every 90 seconds to process 50 elements, and complete the whole import in 7-8 nightly hours, but it seems a very bad compromise: 15 seconds offline every 90.

I cannot use pthreads since this will be a production web server.

I tried to tune Apache increasing memory_limit, max_input_vars, max_execution_time but without any differences: DB using from 450MB to 550MB of RAM and server almost identical.

Linux #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64

Versione software del server: Apache/2.4.10 (Debian)

Versione di PHP: 5.6.35

memory_limit=2048M

max_input_vars=1000000

max_execution_time=600000

MySQL: 5.6.40

Am I facing the problem in the wrong way, or Prestashop’s API are not performant and made for bulk (and performant) product import?

public function batchImportAllProductsFromFile($productsToBeInserted){
    foreach ($productsToBeInserted as $key => $customProduct ) {

        $productIDs = $this->getProductIDsByReference($customProduct->MS_CODMAG);
        if (sizeof($productIDs) == 0) {
            $product = new Product();
        } else if (sizeof($productIDs) == 1) {
            $product = new Product($productIDs[0]);
        } else {
            continue;
        }

        $product->reference = $customProduct->MS_CODMAG;
        $product->name = trim($customProduct->MS_DESCRIZIONE);
        $product->price = $customProduct->MS_PREZZO_1;
        $product->out_of_stock = ($customProduct ->MS_ESAURITO === "S" ? true : false);

        $category = null;

        $msGruppoConverted = $this->buildSubGroupCode($customProduct->MS_GRUPPO, $customProduct->MS_SGRUPPO);

        if ($customProduct->MS_GRUPPO !== 0 && $msGruppoConverted !== 0) {
            $product->id_category = [$customProduct->MS_GRUPPO, $msGruppoConverted];
        } else if ($customProduct->MS_GRUPPO === 0 && $msGruppoConverted !== 0) {
            $product->id_category = [$msGruppoConverted];
        } else if ($customProduct ->MS_GRUPPO !== 0 && $msGruppoConverted === 0) {
            $product->id_category = [$customProduct->MS_GRUPPO];
        }
        try {
            if (sizeof($productIDs) == 0) {
                if ($product->add()) {
                    $product->updateCategories($product->category);
                    $product->addFeatureProductImport($product->id, 1, $customProduct->MS_FAM);
                    //StockAvailable::setQuantity((int)$product->id, 0, $product->quantity, Context::getContext()->shop->id);
                }
            } else if (sizeof($productIDs) == 1) {
                if ($product->update()) {
                    $product->updateCategories($product->category);
                    $alreadySavedFeatures = $product->getFeaturesStatic($productIDs[0]);
                    if (sizeof($alreadySavedFeatures) != 1 || $alreadySavedFeatures[0] != $customProduct->MS_FAM) {
                        $product->deleteProductFeatures();
                        $product->addFeatureProductImport($product->id, 1, $customProduct->MS_FAM);
                    }
                }
            }
        } catch (Exception $e) {
            var_dump("Errore: ", $e, $product);
        }
    }
}

EDIT 22/10/2018:

Upgrading to PHP7.2 and using MariaDB 10.3.10 brought me no changes: timing was still the same.
What did brought benefits was to mount the FS (EXT4) where the DB stores info with option barrers=0 in /etc/fstab: performances improved from 153 to 35 seconds for 500 elements, resulting in circa 18mins totally (was 77).

The problem that remains open is why the system became unresponsive while importing.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    What really solved my problem of Prestashop being stuck while importing products is to move my code from being inside a ModuleAdminController to a WebserviceSpecificManagementInterface: this way the import goes in background without saturing the system.


  2. 0

    You should check that your products needs an update before updating it.

    Here is what I do (Highly simplified) when importing entities on Prestashop:

    <?php

class myProductImporter {
    protected $products;
    protected $products_checksum;

    public function __construct($products) {
        // Your products from the csv file
        $this->products = $products;
        // Here you get an associative array of products references and checksums
        // ex: array('REF01158' => '489f9ze4f4ze9f49ze8', 'REF15616' => '48949844561233132')
        $this->products_checksum = getProductsChecksum();
    }

    public function run() {
        foreach ($this->products as $product) {
            // If the product ref is present in my checksum list, then its an update
            if (isset($this->products_checksum[$product['reference']])) {
                // If the checksum is different, the product needs an update
                if ($this->products_checksum[$product['reference']] != $this->getChecksum($product)) {
                    $this->updateProduct($product);
                }
            // Else it's a new product
            } else {
                $this->addProduct($product);
            }
        }
    }

    protected function updateProduct($product) {
        $PSProduct = getProductByReferebce($product['reference']);
        // Update your product and save its new checksum
    }

    protected function addProduct($product) {
        $PSProduct = new Product();
        // Create the product and save its checksum
    }

    protected function getChecksum($product) {
        // Create a string containing all your product properties
        $checksum = $product['reference'];
        $checksum .= $product['name'];
        $checksum .= $product['description'];
        $checksum .= $product['id_category'];
        return md5($checksum);
    }
}

    When there’s no modification, your import will finish instantly because there’s no queries being processed.

    Concerning the freezes on your installation it seems to be a Docker problem and not linked to Prestashop.

    As mentioned by @bruno-leveque you should consider upgrading to PHP7.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search