Question posted in 
I have a WordPress blog on my LAMP server. I must make 20.000 POST requests to WordPress API in a short period of time to create posts (I make migration from other CMS to WordPress).
For this I run my migration script, which needs to hit https://example.com/wp-json/wp/v2/posts endpoint 20.000 times. After ~1000 requests my server slows down, and the average load goes to 15. When I check top, this is what I get:
5827 www-data 20 0 943612 73000 33424 S 4.7 3.6 0:24.62 apache2
5828 www-data 20 0 943960 79112 38768 S 4.0 3.9 0:16.26 apache2
5861 www-data 20 0 944280 86568 45968 S 4.0 4.3 0:06.47 apache2
6047 www-data 20 0 943692 77288 37292 S 3.7 3.8 0:00.46 apache2
5835 www-data 20 0 942212 85760 47096 S 3.3 4.2 0:05.01 apache2
27086 mysql 20 0 1208472 187212 21124 S 3.0 9.3 18:52.19 mysqld
5863 www-data 20 0 944452 103896 64936 S 2.7 5.1 0:06.97 apache2
5826 www-data 20 0 942604 102272 63264 S 2.0 5.1 0:06.66 apache2
5854 www-data 20 0 944252 84776 46180 S 1.3 4.2 0:06.96 apache2
5967 www-data 20 0 941856 82388 44128 S 1.3 4.1 0:03.98 apache2
6046 www-data 20 0 943692 77476 37356 S 1.3 3.8 0:00.51 apache2
5860 www-data 20 0 942224 84668 46048 S 0.7 4.2 0:06.93 apache2
6059 www-data 20 0 941644 75192 37164 S 0.7 3.7 0:00.20 apache2
When I stop the migration script, those processes are still running and the high server load stays at the same value (around 15). Only thing which helps is sudo apachectl restart. But then I reset apache2, run the migration script again, then after 1000 requests loads go high again.
2
Answers
Apache spawns process to handle requests: assuming you’re using MPM, you can tune it following the official documentation.(https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequestworkers)
Access logs and error logs might help you find out what’s happening.
The problem might lie elsewhere, for example on the database: maybe it’s not tuned to handle the load WP is generating to store the information you want to save calling the APIs.
Moreover, instead of tuning apache/mysql/mariadb/postgreSQL you can try lowering the frequence the script calls the api.
You can try lowring the
You have so many processes because that is how apache has been told to behave. The links @Inc0 provided will give you some clues how you can cap usage. For a faster recovery time you should set a low MaxSpareServers.
If your load is higher than the number of CPU threads, then the kernel will start pre-empting processes – that means your throughput will drop. By how much, I can’t really say. But if this is a one-off exercise, how much time and effort are you prepared to spend getting it to go faster rather than just getting on with load the data / reduced efficiency?
Is this multi-threaded / using curl_multi-* functions ? I wouldn’t expect a single threaded operation to cause so much persistent load. You’d get a lot of clues as to what is going on from mod_status. Regardless, it appears you are not using keepalive in your client (or its deisabled on your server). Enabling this at both ends should have a significant impact on Load / performance.
So this is the XY problem.
WordPress is very slow and resource heavy. Your biggest win would come from bypassing WordPress and writing the data directly to the database. Second best would be using the built-in bulk import function in WordPress (it also exports so you see the data structure). Being WordPress, there are also several third-party plugins you can use (but please uninstall the plugin when you have finished the import.