The Stovebolt Site Forums The Back Office News and Announcements Sorry for the outages

I'm trying to resolve an issue with a file on the server. Since we don't have a test server, I can only work on the live server. This may continue for a while until ai figure out what the cause is.

Now that I've resolved the problem, let me tell you what's going on.

The reason you've been getting timeouts and slow responses is because the internet is literally flooded with bots. Bots are programs that go around the internet doing things. In this case, they're "consuming" content from websites to feed their AI machines. Without first asking if it's OK. And ignoring all the rules that have been established for years. So, I"m blocking them

In the process of doing that, I managed to break the server and had to figure out why. It's now been fixed.

Going forward, our server will still have to process every request the bots make, but now, instead of feeding the bots data from our site, our server will go, Oh, it's you/ Go away. You get nothing.

Hopefully, this will return performance to a reasonable level, but think about this. If we did nothing, we would be forced to pay for a more powerful server to handle the additional load. And the AI bots will pay us nothing for the additional capacity we have to pay for to accommodate them. No Thanks.

I'll leave this up for a few days so everybody can read it. If you want to comment, I've unlocked the thread. If you have questions, I'll answer them.

Thanks for all the searching and researching Paul. As we have lamented many times, this stuff is getting crazy. I can feel my brain letting "botted" ... ... I may need an AI implant.

Are there a lot of bots getting in? I've noticed for a little while that mostly the "Who's Online Now" robots have been "0" or at least a low number. Thanks for the work you're doing!

Even tho they are not listed as bots in the WOL, they are there. I think Paul fixed that designation so the WOL data for that is not accurate. I think that's what he said.

John - We had a lot of META and Google bots that wouldn't get recognized in the bottom bot list for some reason, but if you hover over the "Info" icon to the right of the guest IP addresses, that indicates the "AGENT" behind the IP guest. Those "hidden" bots were pinpointed by looking at the "Info" detail which clearly stated they were bots from those entities. Allowed for identification and subsequent actions to make them go away - YAY!!!

That's because the software isn't updated to recognize modern bots. Trust me, they're there. This is a list of the top ten blocked hosts. Look at the number of connection attempts.
# sh /home/pauls/findblockedhosts.sh
213028 216.73.216.101
205055 216.73.216.126
121110 66.249.66.68
83650 66.249.66.69
44390 66.249.66.70
36572 66.249.73.5
27060 20.171.207.64
18853 66.249.66.71
10909 66.249.73.6
8805 66.249.66.72
The top two are Digital Ocean. The rest are googlebots (they are rate limited not blocked) and the 20 address is azure.net, another ai bot.

So, I finally got so fed up that I did something I haven't done in thirty years. I setup a firewall and blocked the bots at the edge. Now they can't even get to the web server. It's getting really insane. When you get more traffic from bots than you do from actual real website visitors, something is seriously out of kilter.

Sometimes it takes drastic action to fix stuff. If they can't play nice, then hammer them. JMHO.

Paul ...

Is the firewall keeping Peggy and I from accessing the [email protected] email? We have not been able to check that or the [email protected] since Sept. 3.

No. The port for email is unblocked. I will double check everything.