The Stovebolt Site Forums The Back Office IT Short Bus Recent changes to the site

As you are all probably aware, we've been getting hammered with bots recently. The result has been extremely long load times and even inaccessible pages at times. I've been working diligently to determine the cause and rectify the situation. At first, I started blocking entire countries from even accessing the site. But that became a game of whack-a-mole. So, I searched for other solutions.

The first thing I discovered was that we were using default caching on the database which caused long load times under heavy load. I increased the buffer size by a factor of 62, and performance dramatically improved. But I was not satisfied. I kept looking. Working with Grok (AI) I looked at slow queries and discovered that there was a flaw in the UBB software that forced every query to look at every row in the database. (There are well over a million rows now.) So, I fixed the flaw and reported it to the developers. That decreased load times noticeably. I still wasn't satisfied, so I kept looking. I discovered that, despite the fact that queries now only accessed one row instead of millions, there were still delays being introduced because every time a post was read, the topic view count was updated, and that caused lock contentions which resulted in 2 to 3 second query times. So, I fixed that too, and posted the fix to the developer forum.

The forum now loads lickety split. (That's a technical term.) I don't think we'll need to worry about slow load times caused by bots now.

I will send John the bill tomorrow.

UPDATE: Let me see if I can explain this in layman's terms.

Our forum has been using a two lane country road to deliver posts to you. So long as we had country traffic levels, the system worked well and you could login and view the threads, and reply, and enjoy the site. Then automated programs showed up. (bots) They are used for both good and evil purposes, but each bot that accesses our forum adds traffic to that country road. And we have had an increase of thousands and thousands of bots. Eventually, traffic gets so clogged up that it's slow and frustrating, and you begin to wonder if it's worth it.

I've tried blocking the bots, but it's nigh unto impossible. And soon as I blocked them in one place, they showed up in another. So, another solution was needed. In trying to resolve the problem, I discovered three things that were causing the slowdowns.

1) The country road. Instead of a country road, we needed a superhighway. So, I changed one setting for the database, and we went from a two lane road to a 62 lane superhighway. This made a huge difference. Instead of seeing 900% usage of the CPU (computer brain), we were now seeing 5% and 1o%.

2) Once the superhighway was created, the problem was resolved. But, I wanted to see if I could improve on that. So, I took a look at the forum software. I discovered that when you clicked on a post to read it, the software looked at over 1 million posts trying to find the one you asked for. I fixed that so that it only looks for the post you asked for. Since I have modified the software, each time we update, I will have to implement this change again (unless the vendor does the right thing and changes his software.) This reduced CPU usage to 1% to 3%.

3) Still not satisfied that I had found every problem, I continued to look at the forum behavior. I found another bottleneck. When you read a post, the software increments the post view count. While it's doing that, no other post can have its count incremented. It's like you're sailing down this 62 lane superhighway, not a care in the world, and all of a sudden, it narrows down to ten lanes. The result is another bottleneck. I removed that one as well. Now, the CPU usage is down to 0.1% to 0.5%. IOW, the server is sitting there bored to death saying, what else do you want me to do? But there is nothing else, because it's already handled all the assigned tasks.

None of this would have been possible without the use of AI (Grok). Grok can parse and understand computer code faster than I can ask the questions. Grok found and described all these problems based on my questions, and provided me with the solutions (in computer code) to fix them. In this case, I was the bottleneck.

Paul

Kudos for figuring all that out (I have no idea what all that means, but appreciate you doing that.) If John needs help with the bill, let him know there are those of us who will help out.

Paul, WOW! And Thanks!!! Even though I've an OLD introduction to COBOL book somewhere in my stash of history, today I get lost at times and the kids say things like "Dad, please step away from the keyboard"

We are lucky you're on the team.

Ronnie

Thank you Paul! I second “We are lucky you're on the team.”! Great work!

Paul, thank you, for not only fixing the problem, but also doing your best to explain it all to us.

John

I'll add my Thank You to Paul as well and 2nd Kevin's offer to "help" with the bill!!!

Thanks Paul