A customer reported an issue which put me down a rabbit hole and I discovered a strange bug, if you add a domain on port 80 and then another using the same backend on a different port but the same vps/ip you get an odd situation where only 1 is checked and then the wrong one is marked online.
This has been fixed, we have stronger logic now so single backend, multi domain, multi ports works now instead of randomly going offline after around 6 hours.
We did a few more robustness checks on this too to generally improve backend online logic.
Additionally, we were getting load spikes on the HAProxy servers, causing periodic high response times. The root cause has been found, essentially an sqlite db on the haproxy server for stats did not scale well, its only job is to show us stats on the admin side per domain for requests and responce times and actually while it seemed like a great idea at the time, we never expected it to run at this scale so it has 54 million rows and is read multiple times every few minutes to pull data from a timeperiod. This is being stopped and stripped out, we can do it via cli if we need to know.
All in all the HAProxy server will become more responsive and reliable when this is finished, hoping to push to production and clean up today without any live disruption.
TierHive - Hourly VPS - NAT Native - /24 per customer - DE, UK, SG, CA, USA x3, FR, AU, PL, NL FREE tokens on sign up, try before you buy. | Static Hosting Free for life: https://tierhive.com/static-hosting/
A customer reported an issue which put me down a rabbit hole and I discovered a strange bug, if you add a domain on port 80 and then another using the same backend on a different port but the same vps/ip you get an odd situation where only 1 is checked and then the wrong one is marked online.
Thanks for the quick fix. Seems to be working great now
Comments
Few Haproxy updates.
A customer reported an issue which put me down a rabbit hole and I discovered a strange bug, if you add a domain on port 80 and then another using the same backend on a different port but the same vps/ip you get an odd situation where only 1 is checked and then the wrong one is marked online.
This has been fixed, we have stronger logic now so single backend, multi domain, multi ports works now instead of randomly going offline after around 6 hours.
We did a few more robustness checks on this too to generally improve backend online logic.
Additionally, we were getting load spikes on the HAProxy servers, causing periodic high response times. The root cause has been found, essentially an sqlite db on the haproxy server for stats did not scale well, its only job is to show us stats on the admin side per domain for requests and responce times and actually while it seemed like a great idea at the time, we never expected it to run at this scale so it has 54 million rows and is read multiple times every few minutes to pull data from a timeperiod. This is being stopped and stripped out, we can do it via cli if we need to know.
All in all the HAProxy server will become more responsive and reliable when this is finished, hoping to push to production and clean up today without any live disruption.
TierHive - Hourly VPS - NAT Native - /24 per customer - DE, UK, SG, CA, USA x3, FR, AU, PL, NL
FREE tokens on sign up, try before you buy. | Static Hosting Free for life: https://tierhive.com/static-hosting/
Thanks for the quick fix. Seems to be working great now