TierHive General thread | Discuss, Updates, questions | LATEST: Reboots.. reboots everywhere!!

AnthonySmith

Few Haproxy updates.

A customer reported an issue which put me down a rabbit hole and I discovered a strange bug, if you add a domain on port 80 and then another using the same backend on a different port but the same vps/ip you get an odd situation where only 1 is checked and then the wrong one is marked online.

This has been fixed, we have stronger logic now so single backend, multi domain, multi ports works now instead of randomly going offline after around 6 hours.

We did a few more robustness checks on this too to generally improve backend online logic.

Additionally, we were getting load spikes on the HAProxy servers, causing periodic high response times. The root cause has been found, essentially an sqlite db on the haproxy server for stats did not scale well, its only job is to show us stats on the admin side per domain for requests and responce times and actually while it seemed like a great idea at the time, we never expected it to run at this scale so it has 54 million rows and is read multiple times every few minutes to pull data from a timeperiod. This is being stopped and stripped out, we can do it via cli if we need to know.

All in all the HAProxy server will become more responsive and reliable when this is finished, hoping to push to production and clean up today without any live disruption.

Hirose

@AnthonySmith said:
Few Haproxy updates.

A customer reported an issue which put me down a rabbit hole and I discovered a strange bug, if you add a domain on port 80 and then another using the same backend on a different port but the same vps/ip you get an odd situation where only 1 is checked and then the wrong one is marked online.

Thanks for the quick fix. Seems to be working great now

david3

Just curious, is there a way to clear the announcements?

The "keep your recipes up to date" one is always marked new.

btxx

@david3 said:
Just curious, is there a way to clear the announcements?

The "keep your recipes up to date" one is always marked new.

Go under account settings 👍

david3

@btxx said:

@david3 said:
Just curious, is there a way to clear the announcements?

The "keep your recipes up to date" one is always marked new.

Go under account settings 👍

Ah, got it, thanks.

Michal212

Hi @AnthonySmith ,

Could you please confirm if you have already applied the security fix for the Januscape vulnerability (CVE-2026-53359) on your infrastructure/hypervisors?

Thanks

AnthonySmith

@Michal212 said:
Hi @AnthonySmith ,

Could you please confirm if you have already applied the security fix for the Januscape vulnerability (CVE-2026-53359) on your infrastructure/hypervisors?

Thanks

We are going to start the reboots later today and downtime will be as long as it needs to be.

Expect issues.

AnthonySmith

We hit some snags early on, we have been using this as an opportunity in real time to improve things, around 20 changes have gone into production to help the self-healing, did not go quite to plan for the first few nodes, but we stuck with it, as how often do you get an opportunity to reboot your entire infrastructure in an orderly manner eh!

Now by location 6 we are pretty sure we have it nailed down, meaning any unexpected outages, power cycles, hangs, and reboots should self-heal much faster without human intervention in some cases.

I have zero doubt the clever/dumb balance will be restored when we realise we missed something tomorrow when customers open tickets, but, you know, we are in Alpha for a reason.

We are going to continue work on it over the weekend and expect to be all patched up by Monday, by which point we expect at least 2 more CVE's that will require an update and reboot

Seriously, though, if this happens, we are actually considering doing live patching for Debian stable, if this one has been there for 16 years, imagine what else is going to hit....

AnthonySmith

All patched up, might be a bit unpredictable for a few hours while the mesh corrections run in batches