seo Question posted in
Some basic information to get you started with this can be found here.

Best Method for uploading 50,000 Images using File_Put_Contents and File_get_Contents – SEO

I have a CSV with roughly 50,000 images. In the CSV file I have a column for each name of the image and a column for the actual URL address to the image. The code cleans up spaces, apostrophe’s and commas and replaces the spaces with dashes so that the image names will be easier to read as well as more SEO friendly as the original image names are a combination of letters and numbers.

The approach I am using is placing the code on a stylesheet so in order to activate it I go to https://mysite/stylesheet.php. Once the server downloads around 600 – 700 images I eventually end up getting a 500 error.

What would be the best way to download these 50K images onto the server without getting a timeout? No, I do not have direct access to the server, this is a Hostgator Cloud Business setup. I already increased PHP Memory to 1GB from 256MB and that didn’t help at all.

The Code is below:

<?php
$filename = dirname(__FILE__) . DIRECTORY_SEPARATOR . 'photo.csv';
$file = fopen($filename, 'r');
while (($line =fgetcsv($file)) !== FALSE)
{
    $name       =   $line[0];
    $url        =   $line[1];
    $str        =   $name;
    $str        =   str_replace(' '  , '-', strtolower($str)); 
    $str        =   str_replace(''' , '' , $str);
    $str        =   str_replace(',' , '' , $str);   
    $img        =   'mtg/images/'.$str.'.jpg';
    $img_path   =   dirname(__FILE__) . DIRECTORY_SEPARATOR . $img;

    file_put_contents($img_path, file_get_contents($url));   
}
fclose ($file); ?>
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Increasing the limit had basically zero effect due to the mass of images. I ended up taking a different approach using a plugin that automatically renames images as they are uploaded via csv file.


  2. 0

    Could your host have set a limit on the number of requests that can be made within a given duration and host.
    If the number is consistent, stop the transaction on the lower limit(600) for a given duration and continue on the next iteration. You will need to tweak the time and number of files that are processed in each iteration.

     <?php
   $filename = dirname(__FILE__) . DIRECTORY_SEPARATOR . 
     'photo.csv';
   $file = fopen($filename, 'r');
   while (($line =fgetcsv($file)) !== FALSE)
  {
  // Start time
  $startTime = new DateTime();
  //  Pause duration : when do you want the process to pause
 $pauseDuration = 300; // seconds
  // Batch size : how many transactions in the active queue 
   $batchSize = 600;
  // Stop time
  $stopTime = startTime->add(DateInterval(pauseDuration));

 If(startTime == stopTime) { 
     // update the stoptime 
     sleep(pauseDuration); } else { 

   $name       =   $line[0];
   $url        =   $line[1];
   $str        =   $name;
   $str        =   str_replace(' '  , '-', strtolower($str)); 
   $str        =   str_replace(''' , '' , $str);
   $str        =   str_replace(',' , '' , $str);   
   $img        =   'mtg/images/'.$str.'.jpg';
   $img_path   =   dirname(__FILE__) . DIRECTORY_SEPARATOR . 
    $img;
   file_put_contents($img_path, file_get_contents($url));   
    }
   fclose ($file); ?>
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search