As I coninue to work through this problem, I've come to the conclusion that the forum is the problem. Every other page on the website loads instantly. When I run top, it shows extremely high cpu usage for the database server (which is the forum.)
top
top - 15:40:50 up 37 days, 21:50, 2 users, load average: 7.19, 15.83, 27.35
Tasks: 274 total, 3 running, 271 sleeping, 0 stopped, 0 zombie
%Cpu(s): 39.6 us, 39.2 sy, 0.0 ni, 20.9 id, 0.1 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 15718.4 total, 1303.4 free, 1961.1 used, 12454.0 buff/cache
MiB Swap: 4000.0 total, 3809.3 free, 190.7 used. 13370.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3071685 mysql 20 0 4974296 506424 38448 S 516.3 3.1 71:46.41 mysqld
3071874 apache 20 0 765364 56580 19292 R 20.3 0.4 0:15.25 php-fpm
3071973 apache 20 0 743916 37640 19740 S 19.9 0.2 0:10.23 php-fpm
3071958 apache 20 0 739824 33888 20060 S 18.3 0.2 0:11.03 php-fpm
3072413 apache 20 0 764840 56512 20248 R 17.3 0.4 0:10.91 php-fpm
3071965 apache 20 0 745968 39764 19804 S 12.6 0.2 0:13.08 php-fpm
3072414 apache 20 0 739820 33020 19228 S 8.0 0.2 0:09.17 php-fpm
3071127 apache 20 0 741872 36664 20764 S 5.3 0.2 0:37.67 php-fpm
3072405 apache 20 0 739820 33488 19676 S 4.3 0.2 0:09.56 php-fpm
3071875 apache 20 0 733676 27668 20000 S 1.3 0.2 0:14.09 php-fpm
3072408 apache 20 0 737780 31176 19420 S 1.3 0.2 0:09.37 php-fpm
3069386 apache 20 0 2505452 24460 9000 S 0.3 0.2 0:02.43 httpd
3069599 apache 20 0 2505452 25624 9080 S 0.3 0.2 0:02.79 httpd
3069666 apache 20 0 2505452 28176 9144 S 0.3 0.2 0:03.54 httpd
3069873 apache 20 0 2505452 25528 9272 S 0.3 0.2 0:03.07 httpd
3070009 apache 20 0 2505452 23932 9336 S 0.3 0.1 0:02.90 httpd
3070651 root 20 0 264640 5048 3720 S 0.3 0.0 0:07.27 top Mysql (the database) is using 516.3% of CPU???
That is clearly abnormal. So, I did some testing. I closed the forum, stopped and restarted mysql, reopened the forum, and ran top again. Same problem.
Then I closed the forum and ran top. Mysql was using 0.7% of cpu.
So, something is causing mysql to consume tons of CPU. And that drives response times into the stratosphere.
I"ve noticed recently that our pageview counts have been dramatically higher than normal. (4000 to 6000+) I'm now wondering if we're under some sort of denial of service attack. It's either that, or the forum software is the problem. I have a ticket open with the hosting company actively working this problem, and I have a thread started at UBB to discuss it.
