Redis – How to discover why MariaDB crashes?

I use Nextcloud (18 currently). Everything was working fine until this day when I got this message when accessing my cloud via the webbrowser.

Internal Server Error The server encountered an internal error and was unable to complete your request. Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report. More details can be found in the server log.

I am using Docker, or more specific Docker Compose for almost 2 years now. fpm alpine, split into 5 parts: mariadb, redis, app, web, and cron.

version: '3'

services:
  nextcloud_db:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - nextcloud_db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=
    env_file:
      - db.env

  nextcloud_redis:
    image: redis:alpine
    restart: always

  nextcloud_app:
    build: ./app
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - nextcloud:/var/www/html
    environment:
      - MYSQL_HOST=nextcloud_db
      - REDIS_HOST=nextcloud_redis
    env_file:
      - db.env
    depends_on:
      - nextcloud_db
      - nextcloud_redis

  nextcloud_web:
    build: ./web
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - nextcloud:/var/www/html:ro
    environment:
      - VIRTUAL_HOST=
      - LETSENCRYPT_HOST=
    depends_on:
      - nextcloud_app

  nextcloud_cron:
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - nextcloud:/var/www/html
    entrypoint: /cron.sh
    depends_on:
      - nextcloud_db
      - nextcloud_redis

volumes:
  nextcloud_db:
  nextcloud:

The mariadb one has an error currently.

nextcloud_db_1     | 2020-05-15 01:28:44+02:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.13+maria~bionic started.
nextcloud_db_1     | 2020-05-15 01:28:49+02:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
nextcloud_db_1     | 2020-05-15 01:28:49+02:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.13+maria~bionic started.
nextcloud_db_1     | 2020-05-15  1:28:50 0 [Note] mysqld (mysqld 10.4.13-MariaDB-1:10.4.13+maria~bionic) starting as process 1 ...
nextcloud_db_1     | 2020-05-15  1:28:50 0 [Warning] You need to use --log-bin to make --binlog-format work.
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Using Linux native AIO
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Uses event mutexes
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Number of pools: 1
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Using SSE2 crc32 instructions
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Completed initialization of buffer pool
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=8312011756
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Waiting for purge to start
nextcloud_db_1     | 2020-05-15 01:28:52 0x7f010affd700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.13/storage/innobase/include/fut0lst.ic line 67
nextcloud_db_1     | InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
nextcloud_db_1     | InnoDB: We intentionally generate a memory trap.
nextcloud_db_1     | InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
nextcloud_db_1     | InnoDB: If you get repeated assertion failures or crashes, even
nextcloud_db_1     | InnoDB: immediately after the mysqld startup, there may be
nextcloud_db_1     | InnoDB: corruption in the InnoDB tablespace. Please refer to
nextcloud_db_1     | InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
nextcloud_db_1     | InnoDB: about forcing recovery.
nextcloud_db_1     | 200515  1:28:52 [ERROR] mysqld got signal 6 ;
nextcloud_db_1     | This could be because you hit a bug. It is also possible that this binary
nextcloud_db_1     | or one of the libraries it was linked against is corrupt, improperly built,
nextcloud_db_1     | or misconfigured. This error can also be caused by malfunctioning hardware.
nextcloud_db_1     |
nextcloud_db_1     | To report this bug, see https://mariadb.com/kb/en/reporting-bugs
nextcloud_db_1     |
nextcloud_db_1     | We will try our best to scrape up some info that will hopefully help
nextcloud_db_1     | diagnose the problem, but since we have already crashed,
nextcloud_db_1     | something is definitely wrong and this may fail.
nextcloud_db_1     |
nextcloud_db_1     | Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic
nextcloud_db_1     | key_buffer_size=134217728
nextcloud_db_1     | read_buffer_size=2097152
nextcloud_db_1     | max_used_connections=0
nextcloud_db_1     | max_threads=102
nextcloud_db_1     | thread_count=4
nextcloud_db_1     | It is possible that mysqld could use up to
nextcloud_db_1     | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760255 K  bytes of memory
nextcloud_db_1     | Hope that's ok; if not, decrease some variables in the equation.
nextcloud_db_1     |
nextcloud_db_1     | Thread pointer: 0x7f00fc000c08
nextcloud_db_1     | Attempting backtrace. You can use the following information to find out
nextcloud_db_1     | where mysqld died. If you see no messages after this, something went
nextcloud_db_1     | terribly wrong...
nextcloud_db_1     | stack_bottom = 0x7f010affcc08 thread_stack 0x49000
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: 10.4.13 started; log sequence number 8312011765; transaction id 20582626
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: !!! innodb_force_recovery is set to 1 !!!
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] Plugin 'FEEDBACK' is disabled.
nextcloud_db_1     | mysqld(my_print_stacktrace+0x2e)[0x561046981e8e]
nextcloud_db_1     | mysqld(handle_fatal_signal+0x515)[0x5610463fd915]
nextcloud_db_1     | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f0148c84890]
nextcloud_db_1     | /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f01475b8e97]
nextcloud_db_1     | /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f01475ba801]
nextcloud_db_1     | mysqld(+0x57c53c)[0x56104610a53c]
nextcloud_db_1     | mysqld(+0xb35506)[0x5610466c3506]
nextcloud_db_1     | mysqld(+0xb35d5d)[0x5610466c3d5d]
nextcloud_db_1     | mysqld(+0xb39a55)[0x5610466c7a55]
nextcloud_db_1     | mysqld(+0xb1c16e)[0x5610466aa16e]
nextcloud_db_1     | /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f0148c796db]
nextcloud_db_1     | /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f014769b88f]
nextcloud_db_1     |
nextcloud_db_1     | Trying to get some variables.
nextcloud_db_1     | Some pointers may be invalid and cause the dump to abort.
nextcloud_db_1     | Query (0x0):
nextcloud_db_1     | Connection ID (thread ID): 2
nextcloud_db_1     | Status: NOT_KILLED
nextcloud_db_1     |
nextcloud_db_1     | Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
nextcloud_db_1     |
nextcloud_db_1     | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
nextcloud_db_1     | information that should help you find out what is causing the crash.
nextcloud_db_1     | Writing a core file...
nextcloud_db_1     | Working directory at /var/lib/mysql
nextcloud_db_1     | Resource Limits:
nextcloud_db_1     | Limit                     Soft Limit           Hard Limit           Units
nextcloud_db_1     | Max cpu time              unlimited            unlimited            seconds
nextcloud_db_1     | Max file size             unlimited            unlimited            bytes
nextcloud_db_1     | Max data size             unlimited            unlimited            bytes
nextcloud_db_1     | Max stack size            8388608              unlimited            bytes
nextcloud_db_1     | Max core file size        unlimited            unlimited            bytes
nextcloud_db_1     | Max resident set          unlimited            unlimited            bytes
nextcloud_db_1     | Max processes             unlimited            unlimited            processes
nextcloud_db_1     | Max open files            1048576              1048576              files
nextcloud_db_1     | Max locked memory         16777216             16777216             bytes
nextcloud_db_1     | Max address space         unlimited            unlimited            bytes
nextcloud_db_1     | Max file locks            unlimited            unlimited            locks
nextcloud_db_1     | Max pending signals       1545098              1545098              signals
nextcloud_db_1     | Max msgqueue size         819200               819200               bytes
nextcloud_db_1     | Max nice priority         0                    0
nextcloud_db_1     | Max realtime priority     0                    0
nextcloud_db_1     | Max realtime timeout      unlimited            unlimited            us
nextcloud_db_1     | Core pattern: co...
nextcloud_db_1     | Fatal signal 11 while backtracing
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=0
max_threads=102
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760255 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fa650000c08
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fa674ff0c08 thread_stack 0x49000
2020-05-15  0:01:32 0 [Note] InnoDB: 10.4.13 started; log sequence number 8312011765; transaction id 20582626
2020-05-15  0:01:32 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mysqld(my_print_stacktrace+0x2e)[0x55e945e98e8e]
mysqld(handle_fatal_signal+0x515)[0x55e945914915]
2020-05-15  0:01:32 0 [Note] Plugin 'FEEDBACK' is disabled.
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fa6aeddc890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fa6ad710e97]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fa6ad712801]
mysqld(+0x57c53c)[0x55e94562153c]
mysqld(+0xb35506)[0x55e945bda506]
mysqld(+0xb35d5d)[0x55e945bdad5d]
mysqld(+0xb39a55)[0x55e945bdea55]
mysqld(+0xb1c16e)[0x55e945bc116e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fa6aedd16db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fa6ad7f388f]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 4
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            1048576              1048576              files
Max locked memory         16777216             16777216             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       1545098              1545098              signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: co...
Fatal signal 11 while backtracing

This is in the Docker logs. this sounds way too complicated for me and I have no clue.

I have not made any changes to the setup. As I said I am using docker for almost 2 years, with some changes made to the docker-compose.yml file over the time. But this issue just appeared and I haven’t changed anything for about 3 or 4 months.

Is it possible, that it has something to do with a big database? I am using Nextcloud News and do not purge any items because I use them for machine learning data processing. So the database might be a little big.

After amysqldump the oc_news_items table is about 170MB in size and the last automated dump (this morning) doesn’t seem corrupted or something like this, but it has suspiciously 99991 rows. So maybe there might be a problem with over 99999 rows of nextcloud news items in the db? Or maybe only coincidence.