Our old forum facing some hiccups

bikegremlin

Primarily intended as thinking out loud and a personal reminder (until I write a Pulitzer-winning website article ). Of course, any ideas and help are welcome.

Our old forum runs on PHP 5.6 and no one is planning to put in the time to re-write it. So far so good. It's a LiteSpeed hosting environment, so the version used is their "hardened" obsolete version that should be OK...ish in terms of security.

It's hosted on a separate cPanel reseller hosting (sub)account and has been fine for the past two years (since I moved it there).
It uses Cloudflare for both DNS and their free-tier protection.

Today, I got a note from the uptime monitor (HetrixTools) that it's down. I logged into the cPanel account and saw CPU usage at 100%.

No (unusual) error logs or anything (from what can be seen using the reseller client privileges).

I checked on the number of (guest) visitors and saw a number of around 800. That's highly unusual. It should show below 100 at any given time.

I can't find any real-time cPanel and Cloudflare usage stats, so I went with that info (all that's available), and tried the Cloudflare security option "I'm under attack."

That helped immediately. OK - correlation doesn't always mean causation. How can I look further into it?
Again - I'm a cPanel reseller hosting customer and don't expect the provider's tech. support to spend much time over the problem, though I still haven't bothered asked them.

I expect to see the updated visitor number stats by the morning (both under cPanel, and Cloudflare), so I might be able to see a "bump" during the half-hour it took me to realize something strange is happening and activate the CF protection.

It's not a hugely popular forum. And it's not commercial in the least.

To answer the logical question:
The main dev. had gone on to become a fitness instructor and I don't think they'd be able nor willing to re-do and update the code.
And we really like it how it is now, would prefer keeping it to archiving and moving to Vanilla or similar.

Rant over. Carry on.

FAT32

My suggestion would be writing a script to migrate the content to another actively maintained forum software

bikegremlin

@FAT32 said:
My suggestion would be writing a script to migrate the content to another actively maintained forum software

Do you think software (and PHP version) has anything to do with the problems faced today?

Does Vanilla for example have a better anti-bot (if that turns out to have been the problem cause) protection?

FAT32

@bikegremlin said:
Do you think software (and PHP version) has anything to do with the problems faced today?

Does Vanilla for example have a better anti-bot (if that turns out to have been the problem cause) protection?

It is very hard to say... it can be the software itself, the plugins or any vulnerabilities on the PHP version - you can check the log once you have it. From my understanding this forum is written by your previous developer so it is not actively maintained. Therefore, I won't be surprised if there's any vulnerabilities that might have missed out. (Search is usually one of the highest CPU usage in a forum)

I believe Vanilla is just lighter in general (Not really true if you check the network tab), don't have much idea which forum provide the best anti-bot though. (Maybe just disable search on any forum software? )

bikegremlin

@FAT32 said:

@bikegremlin said:
Do you think software (and PHP version) has anything to do with the problems faced today?

Does Vanilla for example have a better anti-bot (if that turns out to have been the problem cause) protection?

It is very hard to say... it can be the software itself, the plugins or any vulnerabilities on the PHP version - you can check the log once you have it. From my understanding this forum is written by your previous developer so it is not actively maintained. Therefore, I won't be surprised if there's any vulnerabilities that might have missed out. (Search is usually one of the highest CPU usage in a forum)

I believe Vanilla is just lighter in general (Not really true if you check the network tab), don't have much idea which forum provide the best anti-bot though. (Maybe just disable search on any forum software? )

It worked fine since we moved it off a really crappy hosting to an OK one.
It's been stable for about two years now. CPU load has been hardly ever over 20 % (mostly within 10 %).

If the problem's been caused by a script, or a hack - moving to a regularly updated solution would make a lot of sense.
Though, as it is now, a custom-made solution, maybe there aren't (m)any readily available scripts to target it?

As it stands now - I still don't really know what had caused the problem. I can guess but I'm not certain.

mikho

Sometimes when I’ve seen sudden spikes because of unusual high number of visitors, it has been a php/db issue.
To many read/write requests from php and the cpu spikes, site is almost unusable.

First time it happened I restarted php-fpm, load dropped instantly, only to slowly rise as the visitors(?) reconnected.

What I found during the search was a couple of IPs crawling the entire site with multiple sessions.

Blocked those 2-3 IPs and problem was gone.
I blame the hosting I chose, it was ok to handle the regular load/traffic but not the sudden increase of visitors.

Falzo

check the logs for user agents. sometimes even google or the likes of semrush majestic and co behave badly.
some senseful entries in a robots.txt could even help?

apart from that... updating php might not solve any of your problems, if there really are bots trying to stupidly throw shit at you or crawlers going nuts. as @mikho pointed out, often banning a few offending IPs already helps. that's where ipsets and or fail2ban can be helpful, though probably not on shared hosting.

AlwaysSkint

@Falzo said: ..or the likes of semrush majestic..

I class these as rogue bots and always block, using mod_sec which is likely unavailable on shared hosting. This leaves IP blocking the (only?) method.

dfroe

@Falzo said:
that's where ipsets and or fail2ban can be helpful, though probably not on shared hosting.

There you probably have .htaccess or the like.

As a last resort blocking IPs by quickly checking and aborting in a very early stage of a base (index.php) or included (config.inc.php) PHP file before the actual processing takes place could be feasible.

bikegremlin

@dfroe said:

@Falzo said:
that's where ipsets and or fail2ban can be helpful, though probably not on shared hosting.

There you probably have .htaccess or the like.

As a last resort blocking IPs by quickly checking and aborting in a very early stage of a base (index.php) or included (config.inc.php) PHP file before the actual processing takes place could be feasible.

I can edit .htaccess at the account level (which should do in this case).