Question posted in 
I have a script in Laravel platform which process thousands of records , and it’s timing out issue. I want to avoid update settings on server like php.ini file.
Is there any way to chunk all records in small pieces and use them in further processing until all records will be used.
I have thousands of records which are getting Price from another server.
So,
1-10 records works fine.
1-100 records works fine.
More than 500 records script stuck and gives error like below.
405 nginx server not allowed
My Route is like :- example.com/updateAllKeywords
Below code is getting keyword to fetch keywords’price from eBay API
$keywordCollection = Keywords::all();
Keywords::chunk(100, function ($keywordCollection) use($request,$keywordCollection,$defaultStartDate,$ignoredItems,$response) {
foreach ($keywordCollection->toArray() as $keyword_result) {
$response = $this->apirequest($request, $keyword_result['id'], $keyword_result, $defaultStartDate, $ignoredItems);
}
});
This is api request function:
/**
* Call Finding API for Active List items
*
* @return API Response
*/
public function apirequest(Request $request, $id, $keyword_result, $defaultStartDate, $ignoredItems) {
try {
/* Request */
$request = new TypesFindItemsAdvancedRequest();
/* 1) default 100 items being displayed without pagination */
$request->paginationInput = new TypesPaginationInput();
$request->paginationInput->entriesPerPage = Config::get('constants.paginationoptions.entries_per_page');
/* 2) ItemFilter for three conditions NEW ,NEWOTHER,USED */
$itemFilterForConditions = new TypesItemFilter();
$itemFilterForConditions->name = 'Condition';
$itemFilterForConditions->value = ['1000', '1500', '3000'];
/* 3) ItemFilter for hide duplicate items */
$itemFilterduplicate = new TypesItemFilter();
$itemFilterduplicate->name = 'HideDuplicateItems';
$itemFilterduplicate->value[] = 'true';
/* 4) pull items before given date / days */
$itemFilterStartDateFrom = new TypesItemFilter();
$itemFilterStartDateFrom->name = 'StartTimeFrom';
$itemFilterStartDateFrom->value[] = date('Y-m-dTH:i:sZ', mktime(0, 0, 0, date('m'), date('d') - $defaultStartDate['defaultStartDate'], date('y')));
/* Exclude Keywords/MPN by set Priced Percentage */
if (!empty($keyword_result['price']) && ($defaultStartDate['exclusionPercentage'] > 0)) {
$percentageAmount = $keyword_result['price'] * ($defaultStartDate['exclusionPercentage'] / 100);
$percentageAmount = floor($percentageAmount).substr($percentageAmount-floor($percentageAmount),1,2+1);
$itemFilterPriced = new TypesItemFilter();
$itemFilterPriced->name = 'MinPrice';
$itemFilterPriced->value[] = $percentageAmount;
}
$itemFilter = (isset($itemFilterPriced)) ? [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom, $itemFilterPriced] : [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom];
$request->itemFilter = $itemFilter;
/* 5) Request KEYWORD/MPN to filter Items Active Lists */
$searchKeyword = (!empty($keyword_result['exclusions'])) ? $keyword_result['keyword'] . " -" . "(" . trim($keyword_result['exclusions']) . ")" : $keyword_result['keyword'];
$request->keywords = $searchKeyword;
/* 6) Request For sorting Items by Lowest To Highest Itemprice Price + Shipping Price */
$request->sortOrder = 'PricePlusShippingLowest';
/* 7) Request For Keyword Search Including the Item Descriptions,Title and Subtitle */
$request->descriptionSearch = ($defaultStartDate['advancedExclusions'] == 1) ? true : false;
$service = $this->index();
/* Response returned by $request */
$response = $this->apiresponse($request, $service, $id, $keyword_result['keyword'], $ignoredItems);
return $response;
} catch (Exception $e) {
echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
exit;
}
}
public function apiresponse($request, $service, $id, $keyword, $ignoredItems) {
try {
$response = $service->findItemsAdvanced($request);
/* fetch all ignored Items */
$newItems = array();
$newOtherItems = array();
$usedItems = array();
$errors = "";
if ($response->ack == "Success") {
$results = $response->searchResult->item;
ob_start();
foreach ($results as $item) {
if ($item->listingInfo->buyItNowAvailable == '1') {
$itemPrice = $item->listingInfo->buyItNowPrice->value;
} elseif (($item->listingInfo->listingType == 'Auction') && ($item->sellingStatus->currentPrice->value == 0)) {
$itemResponse = $this->getItemDetails($item->itemId);
$itemPrice = isset($itemResponse) ? $itemResponse : 0;
} else {
$itemPrice = $item->sellingStatus->currentPrice->value;
}
/* check for ignored items */
if (!in_array($item->itemId, $ignoredItems)) {
/* check for NEW Condition (conditionId = 1000) */
if ($item->condition->conditionId == '1000') {
$newItems[] = array(
'keywordId' => $id,
'conditionId' => (string) $item->condition->conditionId,
'itemId' => $item->itemId,
'title' => $item->title,
'price' => $itemPrice,
'itemURL' => $item->viewItemURL,
'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
);
}
/* check for NEW OTHER Condition (conditionId = 1500) */ elseif ($item->condition->conditionId == '1500') {
$newOtherItems[] = array(
'keywordId' => $id,
'conditionId' => (string) $item->condition->conditionId,
'itemId' => $item->itemId,
'title' => $item->title,
'price' => $itemPrice,
'itemURL' => $item->viewItemURL,
'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
);
}
/* check for USED Condition (conditionId = 3000) */ elseif ($item->condition->conditionId == '3000') {
$usedItems[] = array(
'keywordId' => $id,
'conditionId' => (string) $item->condition->conditionId,
'itemId' => $item->itemId,
'title' => $item->title,
'price' => $itemPrice,
'itemURL' => $item->viewItemURL,
'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
);
}
}
ob_flush();
}
$newItems = array_slice($newItems, 0, 20);
$newOtherItems = array_slice($newOtherItems, 0, 20);
$usedItems = array_slice($usedItems, 0, 20);
$insertItems = array_merge($newItems, $newOtherItems, $usedItems);
/* clean up items */
Items::where('keywordId', $id)->delete();
/* save pulled items */
$items = Items::insert($insertItems);
} elseif ($response->ack == "Failure") {
foreach ($response->errorMessage->error as $error) {
$errors .= "%s: %snn" . $error->severity === EnumsErrorSeverity::C_ERROR ? '<b>Error</b> ' : '<b>Warning</b> ' . $error->message;
}
}
/* save API log fot the keyword */
$this->setapilog($request, $response, $keyword);
if (!empty($errors)) {
echo json_encode(array('ack' => 'Failure', 'msg' => $errors));
exit;
} else {
return true;
}
} catch (Exception $e) {
echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
exit;
}
}
Any information on this would be greatly appreciated. Thanks!
2
Answers
@kinjal jethva
Problem:
You are using Database Query into foreach loop. You are calling your apirequest() methods into foreach loop.
When yourr $keywords array have 500 records.
You are calling
In total you are calling your 1000 database query in one single http request.
Solution:
Calling $defaultStartDate inside your apirequest() function is totally useless.
Instead call it outside and pass value as a parameter. So it will call once.
$keyword_result can be called better way.
This is just a structure of code you can use.
Create this has a job Laravel Queues
Put your function in the handle method.
Dispatch your job from say your controller
This way you are implementing exponential backoff such that as your array increases you increase your wait time to make the next api request.