Question 354732 in Laravel Question posted in
The official Laravel Documentation can be found here.

Efficient bulk data insertion (250k records) in Laravel without exceeding PHP time limit? (Queue, multi-tenancy)

I’m encountering a PHP execution time error while inserting approximately 250k records into a MySQL database using Laravel Eloquent. I’m aiming to achieve this without increasing the overall execution time, considering the following constraints:

  1. Frontend: Angular
  2. Multi-tenancy: Implemented using
    https://tenancyforlaravel.com/
  3. Attempted solutions: Laravel Queue

Below is my code.

Job File Code –

class MedicineData implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, Batchable;

    public $data;

    public function __construct($data)
    {
        $this->data = $data;

    }

    public function handle(): void
    {
        foreach ($this->data as $key => $value) {
            $medicine = new MedicineName();
            $medicine->name = $value['name'];
            $medicine->save();
        }
    }

}

Controller File Code

$chunks = array_chunk($data, 500);
$batch = Bus::batch([])->dispatch();

foreach ($chunks as $key => $value) {
    $batch->add(new MedicineData($value));
}

I am getting an error of Maximum PHP Execution Time. How to resolve this.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I Solve the problem using the below code. But it is taking more than 20 minutes to insert 250K data into my database.

    set_time_limit(WRITE_YOUR_PREFERRED_TIME_HERE_IN_SECONDS);

    I got the solution from another Stack Overflow Post. That post link is - Fatal error: Maximum execution time of 30 seconds exceeded


  2. 0

    In your case, using Laravel queues is a step in the right direction, but you may need to tweak your approach to make it more efficient. You need to some optimizations in your queue logic:

    public function handle(): void
{
    $dataToInsert = [];
    foreach ($this->data as $value) {
        $dataToInsert[] = [
            'name' => $value['name'],
            // other attributes...
        ];
    }
    // use insert method to bulk inserting
    MedicineName::insert($dataToInsert);

    // If your $this->data already formated array than no need to run loop for formating
    // use direct insert on Model
    // MedicineName::insert($this->data);
}

    The insert method is more suitable for bulk inserts and allows you to insert multiple records into the database with a single query.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search