skip to Main Content

I am changing my SQL request to PDO in order to integrate more security against SQL injecting. I was using before procedural msqli and everything was working ok. Now I updated server to PHP 8.2 and I am using PDO and prepared statements.

The BD I am accessing is old MariaDB and has a table with 247 fields most of which are NULL on a single row. (It is sections for a description of mushrooms, but only some of each are really described in a single row (mushrooms)).

Most of the 247 fields are set to mediumtext because they can content up to a couple paragraphs.

My connection script is :

try {

    db = new PDO("mysql:dbname=$dbname;host=$servername;charset=utf8mb4", $username, $password );

    $db->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
    $db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
} catch (Exception $e) {
    echo 'Erreur de connection: ',  $e->getMessage(), "n";
return;
} 

When I select one row of that table in order to echo the not NULL descriptions column, as soon as I execute the statement I run into a full memory. The error is at the stmt->execute() line, not even at the fetchAll line.

$description = 'someTag';

$sql = "SELECT * FROM description WHERE tag = :tag LIMIT 1";

$stmt = $db->prepare($sql);
$stmt->bindParam(':tag', $description, PDO::PARAM_STR);
$stmt->execute();
$result = $stmt->setFetchMode(PDO::FETCH_ASSOC);
$result = $stmt->fetchAll();
$stmt->closeCursor();

The error :

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 262144 bytes) in /home/mycoqueb/public_html/nou.mycoquebec.org/scripts/bd-query.php on line (**the $stmt->execute() line)

I tried to use :

while ($row = $stmt->fetch()) {

Instead of fetchAll() but it changed noting, I used it on every other request from the same page and it changed noting. I flushed variables that were no more useful to NULL on the rest of the page and it changed nothing. I even isolated the script on an empty page and it exceeds the memory limit.

I tried to add this to my connection script and it changed nothing :

db->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

I tried to select only one field instead of * and this worked, but I don’t have an easy way to find NOT NULL columns in the table for a specific tag (row).

I tried to find a SQL statement to restrict my SELECT to NOT NULL columns but I was not successful.

I know I could get add memory_limit = 256M to the phpini file, but that is not what I want since the website has a high usage and for now its on a multihosted server.

The best solution would be to reengineer the MariaDB database using a table with description sections labels and linking only used sections to the main description table for each row, but I have limited access to the db since it is online with the old website and the old website needs to stay online and up to date for 1-2 years after new deployment. I though of creating a daily cron job to do that but I would like to avoid for the moment.

Any Idea how I can get this to work? It was working fine using msqli on php 8.1.

To answer the worries of contributors that think it’s related to another part of the page code, here is the complete code of the isolated test page.

function convert($size)
 {
    $unit=array('b','kb','mb','gb','tb','pb');
    return @round($size/pow(1024,($i=floor(log($size,1024)))),2).' '.$unit[$i];
 }


$servername = "localhost";
$username = "myUser";
$password = "myPass";
$dbname = "myDB";


try {

    $db = new PDO("mysql:dbname=$dbname;host=$servername;charset=utf8mb4", $username, $password );

    $db->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
    $db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

    //$db->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

 } catch (Exception $e) {
    echo 'Erreur de connection: ',  $e->getMessage(), "n";
    return;
}   

$query = (array_key_exists("query",$_REQUEST) ? $_REQUEST["query"] : "");


if($query != ""){

    $tag_desc = 'Aegerita candida';
    // Code works if I limit to "SELECT Titre FROM description WHERE tag = :tag LIMIT 1";
    $sql_description = "SELECT * FROM description WHERE tag = :tag LIMIT 1";

    $stmt = $db->prepare($sql_description);
    $stmt->bindParam(':tag', $tag_desc, PDO::PARAM_STR);
    
    echo convert(memory_get_usage(true));  // returns 2mb
    $stmt->execute();
    $result = $stmt->setFetchMode(PDO::FETCH_ASSOC);
    
    $i = 0;
    
    while ($row = $stmt->fetch()) {

        if($i == 0) {

            ?><table style="border-collapse:collapse;border: 0;table-layout: fixed;color: #063850;background-color: #DDDDDD;text-align: left"><thead><tr style="border: 2px solid #063850;"> <?php
            foreach (array_keys($row) as $key) {
                //print_r($key);
                echo ("<th style='padding:5px;border: 2px solid #063850;position:sticky;top: -2;z-index:10;background-color:#e5a045'>$key</th>");
            }
            echo ("</tr></thead>");
        }

        $i++;

        echo ("<tr style='border: 2px solid #063850'>");
        foreach ($row as $value) {
            echo ("<td style='padding:2px;border: 2px solid #063850'>$value</td>");
            // <img src="$value" height="400" >
        }
        echo ("</tr>");
    }
    
    if ($i > 0) {
        echo ("</table>");
        echo ('<div style="margin-top:400px">&nbsp;</div>');
    } else {
        echo "Aucun résultat";
    }

}

$db = null;

Has mentioned it automatically work if I limit my sql to a couple fields.

The error is still on the stmt->execute() line.

2

Answers


  1. Chosen as BEST ANSWER

    UPDATE :

    I finally followed @JoSSte concept and created two tables and used JOIN SQL request to reproduce the result. I know this is the best way to go for the long run.

    I am still wondering though why am I easily capable of creating the original fetch with MySQLI : $rows = $db->execute_query($sql)->fetch_all(MYSQLI_ASSOC); without creating stress on the memory and I instantly run out of memory with PDO...

    For interest this is the SQL and crohn job for creating and populating the two tables the two new tables from the original.

    SQL :

    CREATE TABLE `RubriquesTitres` (
      `ID` int(11) NOT NULL AUTO_INCREMENT,
      `Titre` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_swedish_ci NOT NULL,
      `Ordre` smallint(6) NOT NULL,
      PRIMARY KEY (`ID`),
      UNIQUE KEY `Nom` (`Titre`) USING BTREE,
      UNIQUE KEY `Ordre` (`Ordre`) USING BTREE
    ) 
    
    CREATE TABLE `RubriqueItems` (
      `ID` int(6) NOT NULL AUTO_INCREMENT,
      `Especes_ID` int(6) NOT NULL COMMENT 'Synchronisé en CronJob avec MyCardex',
      `tag` varchar(60) NOT NULL COMMENT 'Clé externe (MyCardex)',
      `DerniereCorrection` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE current_timestamp(),
      `RubriqueID` smallint(6) NOT NULL,
      `RubriqueTitre` varchar(200) NOT NULL,
      `RubriqueTexte` text CHARACTER SET utf8mb4 COLLATE utf8mb4_swedish_ci NOT NULL,
      PRIMARY KEY (`ID`) USING BTREE,
      KEY `Especes_ID` (`Especes_ID`) USING BTREE,
      KEY `tag` (`tag`) USING BTREE,
      CONSTRAINT `LienTag` FOREIGN KEY (`tag`) REFERENCES `MyCARDEX` (`tag`) ON DELETE CASCADE ON UPDATE CASCADE
    ) 
    

    CRON JOB

    $sql_description = "SELECT * FROM description LIMIT 1501,6000";
    
    $rows = $db->execute_query($sql_description)->fetch_all(MYSQLI_ASSOC);
    
    foreach($rows as $row) {
        $i = 0;
        foreach ($row as $label => $value) {
                // This part of code was run once for creating the original main table with Tag titles
                /*
                $sql = "INSERT INTO `RubriquesTitres` (`Titre`,`Ordre`) VALUES ('$label', " . ($i * 10) . ")";
                $db->execute_query($sql);
                */
            
            // Excluding some main table columns that are not in the tag table
            if ($value != "" && $value != NULL && $label != "ID_D" && $label != "Especes_ID" && $label !== "tag" && $label != "groupe" && $label != "groupeN" && $label != "nomlat" && $label != "nomcourant" && $label != "DerniereCorrection") { 
                $i++;
                $sql = "INSERT INTO `RubriqueItems` (`Especes_ID`,`tag`,`DerniereCorrection`,`RubriqueID`,`RubriqueTitre`,`RubriqueTexte`) VALUES ('" . $row['Especes_ID'] . "', '" . $row['tag'] . "', '" . $row['DerniereCorrection'] . "', '" . getRubriqueID($label) . "', '$label', '" . addslashes($value) . "')";
                
                $ok = $db->execute_query($sql);
                
                echo $row['tag'] . '<br>';
                
            }
    
        }
    }
    

    AND here is some of my retrieve code :

    $sql = "SELECT RubriqueItems.RubriqueTitre AS Titre, RubriqueItems.RubriqueTexte AS Texte
            FROM RubriqueItems
            INNER JOIN RubriquesTitres ON RubriqueItems.RubriqueID = RubriquesTitres.ID
            INNER JOIN MyCARDEX ON RubriqueItems.tag = MyCARDEX.tag
            WHERE RubriqueItems.tag = :tag
            ORDER BY RubriquesTitres.Ordre ASC";
    
    $stmt = $dbPDO->prepare($sql);
    $stmt->bindParam(':tag', $description, PDO::PARAM_STR);
    $stmt->execute();
    $result = $stmt->setFetchMode(PDO::FETCH_ASSOC);
        
    $i = 0;   
    while($rubrique = $stmt->fetch()) {
    
        if ($rubrique['Titre'] != "" && $rubrique['Texte'] != "" && $rubrique['Titre'] != "Titre") { 
            $output .= '<p class="texte-titre2">' . $rubrique['Titre'] . '</p><p id="' . $rubrique['Titre'] . '" class="texte-titre3">' .  ($rubrique['Titre'] == "Références" || $rubrique['Titre'] == "Confirmation moléculaire" ? linkUrlsInTrustedHtml(nl2br(ucfirst($rubrique['Texte'])))  :  ($showGlossaire ? ajouter_glossaire(nl2br(ucfirst($rubrique['Texte'])),$glossaire) : nl2br(ucfirst($rubrique['Texte'])))) . '</p>'; 
            $i++;
        }
    
    }
    
    $stmt->closeCursor();
    

  2. Could it be that the issue is your data model?

    If you have a table that has a lot of nulls, then the data is most likely not normalized.

    Instead of having 256 columns in your table, maybe just have the ones that are always there, and then relate it to a separate table with three fields: foreign key to the main table, a tag field and a value field.

    If you are rewriting your application to USE PDO anyway, some long-term considerations could be included.

    Example table

    Layout like yours (all data in one table):

    id name email someotherdata tag1 tag2 tag3 tag4
    1 Jaques [email protected] blablabla NULL a NULL 100
    2 Allouette [email protected] blébléblé c a NULL 400
    3 Mogens [email protected] blæblæblæ NULL NULL 5000 NULL
    4 Birthe [email protected] bløbløblø NULL a NULL 100

    Suggested Tag-Value table layout

    Main Table

    id name email someotherdata
    1 Jaques [email protected] blablabla
    2 Allouette [email protected] blébléblé
    3 Mogens [email protected] blæblæblæ
    4 Birthe [email protected] bløbløblø

    Tag-Value Table

    id tag value
    1 tag2 a
    1 tag4 100
    2 tag1 c
    2 tag2 a
    2 tag4 400
    3 tag3 5000
    4 tag2 a
    4 tag4 100

    Usage

    Instead of selecting just your row, you WILL need to either join the results together, or make two separate calls. I would suggest two calls, especially since you are rtunning into memory issues.

    SELECT 
      id, 
      name,
      email,
      someotherdata
    FROM
      MainTable
    WHERE
      id = 1;
    
    -- and then the tags
    SELECT 
      id, 
      tag,
      value
    FROM
      TagValueTable
    WHERE
      id = 1; -- reference to the main table
    
    
    

    Advantaages of this normalized approach

    • Less bloat (not so many nulls)
    • Simpler structures
    • REST ready
    • More resilient
      • Add tags without changing the DDL of the database (ALTER)
    • You can still get the original layout by making subselects and transpositions in mysql
    • You can easily make statistics on the tags related to each other

    Disadvantages

    • Complexity of the data model increases
    • You need to do a lot of extra SQL gymnastics to get EXACTLY what you had before
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search