I have a webserver running Plesk and serving websites via "PHP-FPM served by nginx". I notice several downtimes every day, for some of the domains but not all of them. The domains that get down every day are kind of random, they’re not the same every day. At the times that the websites are getting "down", I am noticing high RAM and SWAP usage, mostly from mysqld and php-fpm processes. I am attaching a screenshot below.
What I have tried so far:
-
Processes with their RAM usage
ps -o pid,user,%mem,command ax | sort -b -k3 -r
-
Concurrent connections by IP to reveal possible http based ddos attacks
netstat -anp |grep ‘tcp|udp’ | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort
-
See GET and POST requests on all domains in real time to find anything suspicious
tcpdump -s 0 -A ‘tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354’ -
mysqltuner.pl for suggestions
-
Changed FPM settings and played with them for over a week and finally returned them to default values due to no effect
Nothing seems to reveal the cause of the issue. Also, restarting any services (the fpm service, or mysqld) does not fix the issue.
2
Answers
PHP has the ability to tell you what memory usage the current execution is using, so you could add debug
error_logcalls that and addmemory_get_usage()to the output you passing to the log and then you can narrow down where the huge jump in memory usage is coming from. in the code and you will then have to work out what that code is doing.But stepping out of coding to work out what is causing this, this is why large production systems should not have DB and PHP on a single server. as running both means you will have the memory duplicated for the stuff going into the DB. the memory in PHP for the application to build the Query String, then the MySQL Server reaciving it for processing, 2 seperate servers prevents that duplication of memory usage.
Your WordPress sites are being targeted by bots that periodically attempt to crack the password to the site one letter at a time and randomly switch the site that they target to avoid being detected. This is very common for wordpress sites, especially those that have payment gateways and do not have the wp-login.php file renamed or protected. There are many different plugins that can protect your wp-login page from these types of attacks. I would be very surprised if the cause was something else.
The bots also run on botnets with all different IPs, so they will be able to avoid detection. Try parsing the nginx log file for "wp-login.php" or "wp-admin" or "admin-ajax" which are common attack vectors.