I’ve a small app running on a raspberry Pi. Has been for over 3 years now. It suddently stopped working and I narrowed down the issue to MySQL.
Attempting to start the service seemingly fails. Even when I’ve set the innodb_force_recovery to 6.
I’ve given up on this instance of the data. All I’m trying to do now is to take a dump and then restore this in a new instance. However I’m (obviously) unable to use mysqldump if MySQL isn’t running!
Any advice on how I can recover the data?
ubuntu@ubuntu:/etc/mysql$ cat my.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
[mysqld]
innodb_force_recovery=6
innodb_purge_threads=0
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
ubuntu@ubuntu:/etc/mysql$ /etc/init.d/mysql start
Starting mysql (via systemctl): mysql.service==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'mysql.service'.
Authenticating as: Ubuntu (ubuntu)
Password:
==== AUTHENTICATION COMPLETE ===
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
failed!
ubuntu@ubuntu:/etc/mysql$ systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2023-05-09 21:58:52 UTC; 18s ago
Process: 4125 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=1/FAILURE)
May 09 21:58:51 ubuntu systemd[1]: mysql.service: Control process exited, code=exited, status=1/FAILURE
May 09 21:58:51 ubuntu systemd[1]: mysql.service: Failed with result 'exit-code'.
May 09 21:58:51 ubuntu systemd[1]: Failed to start MySQL Community Server.
May 09 21:58:52 ubuntu systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
May 09 21:58:52 ubuntu systemd[1]: Stopped MySQL Community Server.
May 09 21:58:52 ubuntu systemd[1]: mysql.service: Start request repeated too quickly.
May 09 21:58:52 ubuntu systemd[1]: mysql.service: Failed with result 'exit-code'.
May 09 21:58:52 ubuntu systemd[1]: Failed to start MySQL Community Server.
ubuntu@ubuntu:/etc/mysql$ mysqld start
ubuntu@ubuntu:/etc/mysql$ cat /var/log/mysql/error.log
2023-05-09T22:00:07.501574Z 0 [Warning] [MY-011037] [Server] The CYCLE timer is not available. WAIT events in the performance_schema will not be timed.
2023-05-09T22:00:08.440437Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.1) starting as process 4322
2023-05-09T22:00:08.466499Z 0 [Warning] [MY-010122] [Server] One can only use the --user switch if running as root
2023-05-09T22:00:08Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=f16c68f8f63a58e9eabfa0c5f251618a2ffed724
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2d) [0x1bf2ad2]
/usr/sbin/mysqld(print_fatal_signal(int)+0x253) [0x1099b58]
/usr/sbin/mysqld(handle_fatal_signal+0x71) [0x1099c72]
/lib/arm-linux-gnueabihf/libc.so.6(+0x27590) [0xb6605590]
/usr/sbin/mysqld(memory::Aligned_atomic<long>::Aligned_atomic()+0x4d) [0x19747e6]
/usr/sbin/mysqld(Delegate::Delegate(unsigned int)+0x39) [0x1974a3e]
/usr/sbin/mysqld(delegates_init()+0x37) [0x1974b80]
/usr/sbin/mysqld(+0x9fb69c) [0xef269c]
/usr/sbin/mysqld(mysqld_main(int, char**)+0x1f73) [0xef8d08]
/lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x97) [0xb65f55a4]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
ubuntu@ubuntu:/etc/mysql$ mysql -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
2
Answers
Issue has been resolved. As hinted by @Wiss_g, there was an issue with the armhf build for mysql on ubuntu. Additional details can be found at this link: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/2019203?comments=all
General gist is that it was resolved in mysql-8.0 - 8.0.33-0ubuntu0.XX.YY.Z
Must be armhf related.
I’m having the exact same issue also on a raspberry pi.
My raspberry pi (ubuntu server 20.04) has been running mysql server perfectly well for 2-3 years, then suddenly crashed a few days ago.