As I coninue to work through this problem, I've come to the conclusion that the forum is the problem. Every other page on the website loads instantly. When I run top, it shows extremely high cpu usage for the database server (which is the forum.)
Code
 top

top - 15:40:50 up 37 days, 21:50,  2 users,  load average: 7.19, 15.83, 27.35
Tasks: 274 total,   3 running, 271 sleeping,   0 stopped,   0 zombie
%Cpu(s): 39.6 us, 39.2 sy,  0.0 ni, 20.9 id,  0.1 wa,  0.1 hi,  0.1 si,  0.0 st
MiB Mem :  15718.4 total,   1303.4 free,   1961.1 used,  12454.0 buff/cache
MiB Swap:   4000.0 total,   3809.3 free,    190.7 used.  13370.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                   
3071685 mysql     20   0 4974296 506424  38448 S 516.3   3.1  71:46.41 mysqld                                                    
3071874 apache    20   0  765364  56580  19292 R  20.3   0.4   0:15.25 php-fpm                                                   
3071973 apache    20   0  743916  37640  19740 S  19.9   0.2   0:10.23 php-fpm                                                   
3071958 apache    20   0  739824  33888  20060 S  18.3   0.2   0:11.03 php-fpm                                                   
3072413 apache    20   0  764840  56512  20248 R  17.3   0.4   0:10.91 php-fpm                                                   
3071965 apache    20   0  745968  39764  19804 S  12.6   0.2   0:13.08 php-fpm                                                   
3072414 apache    20   0  739820  33020  19228 S   8.0   0.2   0:09.17 php-fpm                                                   
3071127 apache    20   0  741872  36664  20764 S   5.3   0.2   0:37.67 php-fpm                                                   
3072405 apache    20   0  739820  33488  19676 S   4.3   0.2   0:09.56 php-fpm                                                   
3071875 apache    20   0  733676  27668  20000 S   1.3   0.2   0:14.09 php-fpm                                                   
3072408 apache    20   0  737780  31176  19420 S   1.3   0.2   0:09.37 php-fpm                                                   
3069386 apache    20   0 2505452  24460   9000 S   0.3   0.2   0:02.43 httpd                                                     
3069599 apache    20   0 2505452  25624   9080 S   0.3   0.2   0:02.79 httpd                                                     
3069666 apache    20   0 2505452  28176   9144 S   0.3   0.2   0:03.54 httpd                                                     
3069873 apache    20   0 2505452  25528   9272 S   0.3   0.2   0:03.07 httpd                                                     
3070009 apache    20   0 2505452  23932   9336 S   0.3   0.1   0:02.90 httpd                                                     
3070651 root      20   0  264640   5048   3720 S   0.3   0.0   0:07.27 top
Mysql (the database) is using 516.3% of CPU???

That is clearly abnormal. So, I did some testing. I closed the forum, stopped and restarted mysql, reopened the forum, and ran top again. Same problem.

Then I closed the forum and ran top. Mysql was using 0.7% of cpu.

So, something is causing mysql to consume tons of CPU. And that drives response times into the stratosphere.

I"ve noticed recently that our pageview counts have been dramatically higher than normal. (4000 to 6000+) I'm now wondering if we're under some sort of denial of service attack. It's either that, or the forum software is the problem. I have a ticket open with the hosting company actively working this problem, and I have a thread started at UBB to discuss it.

Paul Schmehl CI 6
geek@stovebolt.com
Stovebolt Staff: Geek