I have 2 servers installed in 2 differents localisation, with a synchronisation files managed by heartbeat + drbd. They are running since few months and the date of the last main server reboot is : 2022-12-07 15:44
This night on the main server, the Mariadb service has been stopped. See below the logs :
Dec 18 06:50:14 nodegra systemd[1]: Starting Daily apt upgrade and clean activities...
Dec 18 06:50:20 nodegra pacemaker-controld[1131]: notice: High CPU load detected: 1.370000
Dec 18 06:50:50 nodegra pacemaker-controld[1131]: notice: High CPU load detected: 1.620000
Dec 18 06:51:20 nodegra pacemaker-controld[1131]: notice: High CPU load detected: 1.980000
Dec 18 06:51:50 nodegra pacemaker-controld[1131]: notice: High CPU load detected: 2.120000
Dec 18 06:52:20 nodegra pacemaker-controld[1131]: notice: High CPU load detected: 2.210000
Dec 18 06:52:36 nodegra systemd[1]: Stopping MariaDB 10.5.15 database server...
Dec 18 06:52:36 nodegra mariadbd[1949]: 2022-12-18 6:52:36 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
Dec 18 06:52:36 nodegra mariadbd[1949]: 2022-12-18 6:52:36 0 [Note] Event Scheduler: Purging the queue. 0 events
Dec 18 06:52:36 nodegra kayelles.service[1967382]: Warning: got packets out of order. Expected 2 but received 1
Dec 18 06:52:37 nodegra kayelles.service[1967382]: Warning: got packets out of order. Expected 2 but received 1
Dec 18 06:52:37 nodegra kayelles.service[1967382]: Warning: got packets out of order. Expected 2 but received 1
Dec 18 06:52:37 nodegra kayelles.service[1967382]: Warning: got packets out of order. Expected 2 but received 1
Dec 18 06:52:37 nodegra kayelles.service[1967382]: Warning: got packets out of order. Expected 15 but received 1
Dec 18 06:52:37 nodegra mariadbd[1949]: 2022-12-18 6:52:37 0 [Note] InnoDB: FTS optimize thread exiting.
Dec 18 06:52:37 nodegra kayelles.service[1967382]: db error Error: Connection was killed
I don’t find/understand the reason of the MariaDb service stop.
Is there a way to find the requester ?
2
Answers
Ths for your reply. Heartbeat log are empty fot this day and there is nothing special about in the pacemaker log :
I don't know what I have to check on more or that I have to set to avoid this issue again.
"initiated by: unknown" is an indicator that it wasn’t a user doing
SHUTDOWN
and was as a result of a signal.Base on the close occurrence of the packmaker-control notice message I suspect it may have been the cause. Look closer at your pacemaker/heartbeat logs.