@AnthonySmith said:
Yep, the thing is with an artificial load and no way to tell how it will be used in advance and people also knowing it is a test there is no way to get concrete results.
It will however have a control group that will not know they are a control group. So some insight can be gained.
Additionally it will test an extreme resource density model that I have had in my head as an idea for a while that I am certain some hosts use but I have never had the balls or lack of morals to try in practice.
Also, it’s fun for the whole community.
Frankly, there is no way to get generalizable results because in the first place, you will not get a probability sample for your participants. So, yes, this is a fun test and the results will be indicative but not predictive.
Would it make sense to run a speed/stress test - was that intended?
For comparison of how performance changes depending on whether a server is oversold, or not.
The point is not to stress test. That's easy. The goal is to simulate a "natural" experiment (in a true natural experiment, the participants don't know they are part of the experiment so this proposal lies midway between a proper natural experiment and a controlled experiment) to see if participants notice a difference that leads them to suspect a server has been oversold.
When you stress test, you are basically forcing maximum performance out of the machine for the entire duration, but for oversold servers, you probably won't get to stress test levels all the time (well, you can reach that state if horribly oversold enough, but most providers seek to oversell a bit (understandable for more profit). The key question is what is the distribution of different users' usage? Does it follow a normal distribution, or is it skewed to the left or to the right?
If it is a left skew, you can oversell more and not worry. If it is a right skew, better not. We don't know, and that's what the experiment is about.
@AnthonySmith said:
That is all, no one is trying to get a paper peer-reviewed or published in a recognised journal, no one is saying there will not be areas of weakness in the method due to the scale, it is what it is and it is very interesting and maybe it poses more questions than is answers but that is not a bad thing either.
Not trying to get published in a peer-reviewed journal doesn't mean not treating the study design seriously. You don't want to waste time or money. Just saying.
@bikegremlin said: I still don't see how a comparative performance test can harm anything. It would only give more info, that could be useful in judging performance reducing effect of overselling more objectively.
Without it, it's like avoiding to use a thermometer in comparing whether cellar is cooler than an air conditioned room. Sure - it's great to know the subjective impression. But why deliberately disregard any measurement?
I have no issue with it in general but there is no doubt that the sold to spec server will perform better.
The test is a perception test at its core.
For example, we all know a v6 ford engine with a turbo is going to better than one without a turbo but if you want to find out if people think that the one without the turbo has an acceptable performance based on their past car experience as regular drivers of multiple vehicles then you don't put it on a rolling road to show performance metrics first as that will skew perception and perception is reality.
This is why I said from the getgo if this is done, and you buy it, if your only reason for buying is to find out which server is which then you missed the point.
Again though, I would expect to do the reveal at about 60 days in of a 90-day test so at that point comparative benchmarks could be a great addition to the test.
This reminds me of the tickets I get during black Friday like "I know its a black Friday special but I expect better performance". Upon checking the host node it becomes obvious that 30 customers are benchmarking at the same time with zero awareness of each other.
If 30 users benchmark on the undersold node it will get worse results than if no one is benchmarking on the oversold one
As such personally I put zero store by synthetic generic benchmarks on a VPS because you are doing it blind, so yes it does no harm if done at the appropriate time which is not at the start of the experiment and they would be better off being done by me at a time of known average load after 60 days has established that baseline.
If you want and are really interested I would be happy to have you keep that part honest by overseeing the method and execution.
I understand your point - but.
Not arguing - just trying to explain my point of view - if it's of any practical use for your test.
Tests I had in mind measure sort of a non-critical load (for a somewhat decent hosting with caching set properly). If there are hick-ups with those - then I conclude that server is not acceptably good. If there are large performance differences - same conclusion on which server is worse (not good enough).
Because, from my experience, when relying on "subjective testing", even Bluehost shared hosting is good - until it isn't - which sometimes takes months to become apparent (just discussing the performance, not tech. support, or security). I've been a happy EIG user for years, until I noticed problems - been measuring ever since, because it's the fastest way to compare and see if a hosting is (any) good.
Of course, I have almost no experience with VPS-s, so this could all be wrong. Maybe with VPS-s differences are more noticeable and more gradual.
EDIT: the idea is for you (or one, designated user) to do the tests - not for everyone to be doing stress tests - then it would not be goood.
Comments
The point is not to stress test. That's easy. The goal is to simulate a "natural" experiment (in a true natural experiment, the participants don't know they are part of the experiment so this proposal lies midway between a proper natural experiment and a controlled experiment) to see if participants notice a difference that leads them to suspect a server has been oversold.
When you stress test, you are basically forcing maximum performance out of the machine for the entire duration, but for oversold servers, you probably won't get to stress test levels all the time (well, you can reach that state if horribly oversold enough, but most providers seek to oversell a bit (understandable for more profit). The key question is what is the distribution of different users' usage? Does it follow a normal distribution, or is it skewed to the left or to the right?
If it is a left skew, you can oversell more and not worry. If it is a right skew, better not. We don't know, and that's what the experiment is about.
Deals and Reviews: LowEndBoxes Review | Avoid dodgy providers with The LEBRE Whitelist | Free hosting (with conditions): Evolution-Host, NanoKVM, FreeMach, ServedEZ | Get expert copyediting and copywriting help at The Write Flow
Not trying to get published in a peer-reviewed journal doesn't mean not treating the study design seriously. You don't want to waste time or money. Just saying.
Deals and Reviews: LowEndBoxes Review | Avoid dodgy providers with The LEBRE Whitelist | Free hosting (with conditions): Evolution-Host, NanoKVM, FreeMach, ServedEZ | Get expert copyediting and copywriting help at The Write Flow
I understand your point - but.
Not arguing - just trying to explain my point of view - if it's of any practical use for your test.
Tests I had in mind measure sort of a non-critical load (for a somewhat decent hosting with caching set properly). If there are hick-ups with those - then I conclude that server is not acceptably good. If there are large performance differences - same conclusion on which server is worse (not good enough).
Because, from my experience, when relying on "subjective testing", even Bluehost shared hosting is good - until it isn't - which sometimes takes months to become apparent (just discussing the performance, not tech. support, or security). I've been a happy EIG user for years, until I noticed problems - been measuring ever since, because it's the fastest way to compare and see if a hosting is (any) good.
Of course, I have almost no experience with VPS-s, so this could all be wrong. Maybe with VPS-s differences are more noticeable and more gradual.
EDIT: the idea is for you (or one, designated user) to do the tests - not for everyone to be doing stress tests - then it would not be goood.
🔧 BikeGremlin guides & resources
Soooooo... where are we?
Never got any suitable hardware offers so it was not possible. It’s one that may happen in the future.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
We’re still happy to provide hardware to whatever specs necessary