An interesting tech test, looking for sponsors or pledges from interested people. [Industry Secrets]

InceptionHosting

origin of this thread: https://talk.lowendspirit.com/discussion/comment/30849/#Comment_30849

Background and questions to answer.

There has always been a question around overselling, how much is, ok, what methods are used, what is oversold, what are the limits, will it still be usable if oversold.

To not put this on any specific host there has been questions in the past on how hosts can sell so much ram/specs for so little and still exist years later, one of the suspicions has always been using large or just fast ssd arrays as swap and selling swap as Ram.

The plan

To get 2 identical physical servers and run them for 2 or 3 months, 1 will be configured for significant overselling, the other will not be oversold.

Customers will not know which server they are on, that will only be revealed at the end of the experiment.

Services will be KVM based.

All technical details and methodologies will be documented and posted here and on the main site.

The expectations

As customers will only be able to buy one of these i.e. they will only have a server on 1 node so will have no way to do any direct comparison themselves they will not know which node they are on and service quality will seem acceptable even if on the highly oversold node.

The technical details

I would be looking to use 2 nodes with somewhere close to the following specs.

• 12 Threads minimum
• 32GB Ram minimum
• 4 x SATA SSD's minimum (NVME preferred or 2 x NVME + 2 x SSD)
• 1 x /24 IP range.

1 node will be configured as standard. LVM, 30 of the 32GB RAM Sold, Raid 1 minimum (maybe 10 depends on specs)

1 node will be configured to use ram, lower priority large SSD SWAP spaces, Thin provisioned disks with a 4x oversell i.e. 128GB Ram sold.

Plan for selling.

There will be 1 package/plan based on common requirements along the lines of:

• 2 vCPU
• 2 GB RAM
• 30GB Disk space
• 50 GB BW (just to avoid potential abuse)

I will figure out what the lowest price can be so it does not cost money to sell it (fee's etc) I expect this to be around €0.50 for the entire term which I hope to be 2 months minimum, possibly 3.

The reason for charging is that I don't want to make it free so participants can be verified to some degree as the potential for abuse is rife and to manually provision based on requests is too time-consuming.

What is needed to make this happen.

Option 1. In an ideal world a host that provides dedicated servers sponsors this, please tag any hosts you think may be interested in participating obviously they will be heavily credited through the documenting of this which I expect to be multiple parts and I expect to lead to sales as I will essentially be providing a blueprint for becoming a host on your network with maximum profitability per node.

Option 2. As above but the host provides the servers at a cost which if I can cover myself I will but may ask for pledges from the community.

Option 3. We all collectively agree on a host and chip in at full price to spread the cost.

The requirements repeated in case you skipped that above:

• 12 Threads minimum
• 32GB Ram minimum
• 4 x SATA SSD's minimum (NVME preferred or 2 x NVME + 2 x SSD)
• 1 x /24 IP range.
• Both servers must be in the same physical location and connected to the same physical switch/core to avoid any external factors influencing perceived performance.

Control and keeping it honest.

Aside from 1 of the nodes not being oversold at all, I will be inviting other experienced community members to audit and verify the setup and provide comments, ideally @Francisco if he is willing as he is a well known trusted technical resource but not limited to 1 other host so any others are welcome to be involved.

If this happens what should I know?

1. If you buy a VPS assume your VPS data is not private, others will need access to the servers to validate the experiment.
2. Assume it will not be stable, this is an experiment that may need to be tweaked.
3. If 1 node needs to be rebooted both will be rebooted so as not to give away which is which.
4. If this happens and If you buy one and think a sentence that goes anything like "I know it's only €0.50 and/or I know it is an experiment BUT" this is not for you.
5. There will be no support, NO SUPPORT.
6. Abusers accept they will be named and shamed.
7. Linux/templates only for simplicity and to avoid control panel based limitations artificially impacting this.

---

Any questions, thoughts, words of support?

teamacc

Given the overselling, you'd have about 4x more users on the oversold node than on the non oversold one. Can you still guarantee a 50 50 chance on what node a user lands on, and does that matter?

teamacc

Also, @Clouvider the above might be a nice way to kickstart some of your new locations with respect to resellers?

InceptionHosting

@teamacc said:
Given the overselling, you'd have about 4x more users on the oversold node than on the non oversold one. Can you still guarantee a 50 50 chance on what node a user lands on, and does that matter?

I mean obviously not, all that matters is that you dont know as the buyer which one you are on.

I suppose I should also add generally that if your entire motivation for taking part is to find out yourself which one you are on and publish that before the end of the experiment, you kind of missed the point although if you figure it out anyway then that is fine.

I have no doubt that some tech-savvy people will be able to work it out, I could, I suppose it is also fine to speculate publicly, as long as you understand I won't reveal or answer questions around that until the end.

I expect that there will be a difference is service quality, that is not the point, the point is would you still buy it if the price is right.

I should have said the regular plan price will be positioned at €1.75 p/month

deepak_leb

Sounds cool. ? Considering your requirements, I believe am not a tech savvy.
Anyway waiting for your result of this Experiment

cybertech

Excited for results, and hope contabo sponsors this experiment

AlexanderM

First post here We're more than happy to sponsor the /24 subnet!

foxone

@cybertech said:
Excited for results, and hope contabo sponsors this experiment

Lmao

InceptionHosting

@AlexanderM said:
First post here We're more than happy to sponsor the /24 subnet!

Brilliant, that's a big part of the puzzle.

Brendan

FlowVPS is also willing to sponsor a /24 Know you only need one but will put the offer out there anyway.

wdmg

Not a hosting provider, but also happy to sponsor a /24 or chip in money towards operational costs.

MrPsycho

Count me in. Willing to go even for 3rd option (if EU).

InceptionHosting

looks like the IP space is covered, really appreciated, now we just need to secure the hardware for ideally 3 months.

dahartigan

Nice idea Ant! I'm interested to see how this plays out, and will buy one once I can think of a meaningful use for it. My only concern would be that the usage patterns you'll see on these would not reflect real-world results (will they all just idle or will everyone have geekbench on a hourly cron or something crazy? Lol)

I suspect that an "oversold" node could be as close to non-oversold in terms of performance, where the node is monitored and proactively kept in good health. To the average user, it's going to be virtually indistinguishable.

You'd pretty much have to use alphawootracks as the benchmark for horrible performance imo lol

FAT32

Interesting experiment Let me know if you need a really noisy neighbour - but will this consider as an abuse?

InceptionHosting

@dartagnan @FAT32 yeah the very nature of this experiment will nurture none conventional use, I was thinking about that while writing the initial post.

The bottom line is that I can't really control that but I decided that both servers will be subject to the same unnatural use so it should balance out.

I am also thinking I may pick a bunch of guest servers at random for the last month/phase and migrate them to the other nodes as that may make it even more interesting.

Any hey, if it really works out I might even end up keeping the servers and offering a ridiculous price to keep them.

InceptionHosting

@dahartigan said: You'd pretty much have to use alphawootracks as the benchmark for horrible performance imo lol

I think that was caused by them simply overselling rather than applying some intelligent design to it.

FAT32

@AnthonySmith said:
Any hey, if it really works out I might even end up keeping the servers and offering a ridiculous price to keep them.

Just to confirm, does ridiculous means cheap?

InceptionHosting

@FAT32 said: Just to confirm, does ridiculous means cheap

yeah, cheaper than contabo

vyas

Cool idea!

50 GB BW (just to avoid potential abuse)

Interesting

havoc

Ambitious & interesting. I like it.

I'd drop the spec's to 1 vcpu & oversell that too? Presumably people willing to oversell aren't exactly shy once they've embraced the dark side?

I think it'll be quite dependent on luck of the draw tbh on workloads. i.e. If the oversold one gets allocated more heavy idlers then it'll slant it. So would collect resource usage over time so the extent to which the "over" actually gets used

InceptionHosting

yeah, I should have added some detail around that, the main thing here is the disk and Ram overselling, every VPS host oversells CPU, it is such a given I did not think to even mention it.

sanvit

If you oversell 4x, it would be 64 customers (128vCPU) on a 12thread machine. Wouldn't CPU power also be an issue (assuming that you are mostly considering on RAM usage)?

InceptionHosting

@sanvit said:
If you oversell 4x, it would be 64 customers (128vCPU) on a 12thread machine. Wouldn't CPU power also be an issue (assuming that you are mostly considering on RAM usage)?

Maybe I should have said "up to 4x" there is no absolute formula without a customer use crystal ball however the truth is that most customers barley touch even 10% of a core on avg.

But that is part of the fun of this experiment, maybe I will get 3x maybe I will get 5x but I will push it to the limits and it will be a very interesting analysis afterwards.

WSCallum

It'll certainly be interesting to see the results. It's worth considering a lot of these scenarios are with providers who provide high RAM packages on nodes with like 1TB of RAM, which in itself makes overselling less noticable as the vast majority don't actually use their resources - They just buy 60GB RAM plans because its cheap and cheerful for what it is and then probably use 10GB at the most half of the time.

Looking forward to the results of this!

sanvit

@AnthonySmith said:

@sanvit said:
If you oversell 4x, it would be 64 customers (128vCPU) on a 12thread machine. Wouldn't CPU power also be an issue (assuming that you are mostly considering on RAM usage)?

Maybe I should have said "up to 4x" there is no absolute formula without a customer use crystal ball however the truth is that most customers barley touch even 10% of a core on avg.

But that is part of the fun of this experiment, maybe I will get 3x maybe I will get 5x but I will push it to the limits and it will be a very interesting analysis afterwards.

makes more sense! Thanks for the reply, and really interested on the result

havoc

Could also be an interesting business offering. i.e. VMs with very good "specs" but it's specifically declared as heavily oversold. For some of my use cases that would be useful. e.g. I needed a high RAM one because I couldn't get gitlab to install on smaller ones...but obviously the definition of "RAM" is flexible...

Interestingly Google just made an "overselling" announcement of it's own - very similar type thinking:

https://cloud.google.com/compute/docs/nodes/overcommitting-cpus-sole-tenant-vms

poisson

Oversold to 4X idling customers will just skew the results. Just saying.

bugrakoc

Definitely an interesting experiment. I'll happily buy and idle one! I just hope enough people puts some kind of load on it though, since we are notorious for idling.

lentro

I'm interested in helping cover the costs. Great experiment, and I'm excited to see what the results are!

ReadyDedis

I'll be cool with sponsoring 2 units of 2 x Dual E5-2620v2 - 32GB RAM - 4x240GB SSD ( or 2x500GB SSD )

Not_Oles

@AnthonySmith said: Any questions, thoughts, words of support?

Please wait a moment. This is a really super fun idea! But maybe there are as yet insufficiently considered questions of experimental data and statistical analysis?

First, experimental data: how do the control and experimental groups report their perceptions of server performance? In other words, what data will be collected and analyzed? For example, do the users fill out a questionnaire several times during the experiment and again at the end? Or something else?

Second, statistical analysis: what size does the study need to be in order for the results to be statistically significant? I remember that the famous lady tasting tea experiment is subject to criticism that there weren't enough tests.

I know very little about all this, but I am guessing it might be a great idea, before starting, to think carefully about experimental data and statistical analysis.

Hope this helps!

InceptionHosting

feel free to propose some questions, I was thinking of a questionnaire at 50% capacity and then again at the end.

There are 3 bit of knowledge I want to gain and share from this.

1. Is the method viable, what is the breaking point.
2. is the end-user experience compatible to an under-provisioned node.
3. Even if the overselling is noticeable as a comparison is the service still viable and worth paying for if it is cheap.

seriesn

Not a dedi provider but will be happy to foot the bill for 1 Dedi

bikegremlin

Overselling is not the same as overloading.

It would be interesting to have 4 test servers.
One not oversold.
Second "conservatively" oversold.
Third "normally" oversold.
Fourth "optimistically" oversold.

InceptionHosting

@bikegremlin said:
Overselling is not the same as overloading.

It would be interesting to have 4 test servers.
One not oversold.
Second "conservatively" oversold.
Third "normally" oversold.
Fourth "optimistically" oversold.

Well that's an expensive experiment

also if you just do a vanilla install and throw 256GB of VPS at a server with 64GB Ram.... you are going to have a bad day.

If you tweak the bejesus out of it and manage the ram compression, swap, create a KSM profile to fit the use then you might have a hard day but you might get through it

serv_ee

@AnthonySmith said:
looks like the IP space is covered, really appreciated, now we just need to secure the hardware for ideally 3 months.

There are some 12 core opterons available for pretty cheap right now if you maybe want to buy two servers outright and just do whatever you want with them later on.

comi

This is so cool and @AntGoldFish is best animator 2020 confirmed, but my nerdy side agrees with @poisson and @Not_Oles - this cannot claim to prove anything, as any result you get is just happened to be so.

Do you wish to show some ppl it's ok to oversell a sane ratio? Will they learn though? Do you want to find out how much is ok? The answer is known: "it depends"

That being said I would purchase a 3 month participation for anything less than $7, if only for lulz.

InceptionHosting

@comi I understand what you are saying, I am not trying to prove an ironclad point, this is a technical exercise with observers that shows the art of the possible and will perhaps surprise some people who believe that 10% over commit = turd service and to push the boundaries to find a level.

edit: Also this is to illustrate that it is entirely possible and likely that these services that have been selling 8GB Ram servers for stupidly low prices e.g. 3.99 p/m for the past 5 years or so have probably been simply selling ssd swap space as ram and to see if properly tuned that is possible without hurting the user experience much (if cheap enough).

With NVMe hitting insane speeds now that is likely even more possible.

vimalware

I always thought a provider could only get away with this sort of setup with a very high installed RAM on host. 256G-1024G (let's say 128G as a minimum)

Hetzner server bidding keeps having those 12T, 256GB boxes for under 70EUR/mo. Subnets would be extra.

Also 2GB ram is sort of the new 1GB. I think a lot of better oversell planning could be done with 4 and 8GB RAM guests with lots of idle ram. OFC, Don't hand out vCPU like candy .

App examples:
I know that Gitlab-fresh idles close to 3GB (not including disk pagecache for repos ) .
Zulip recommends 4GB.

bikegremlin

My 2c - not (claiming to be) an expert - have that in mind, but if it's of any use:

Overselling is a rather reasonable way to sell resources. Because many websites (not sure about other on-line applications) are idle at times, more or less. As long as overselling is well balanced, it allows for a lot lower price (and better use of the resources), with very little performance "penalties". In my opinion, it makes little sense to not oversell.

Overloading is a different thing. Overdo overselling, and there are problems. Suppose it boils down to experience and good monitoring by the hosting provider (and calculating some margin of error, or resource use spikes) to make sure that overselling doesn't turn into overloading. Good (especially low end) providers get this right, most of the time. Allowing them to offer affordable prices, with no performance problems (cutting down on overselling means prices must go up, while overdoing it means the performance suffers).

For testing and comparison, I use a free, primitive method (my testing method). TL/DR - two identical WordPress websites, one on each server, then using Octoperf to simulate 50 simultaneous visitors browsing them, and using manual GTmetrix testing. Method is primitive, precision varies up to 14%, but it's the best I can do (on my budget and knowledge levels). Running tests 3 times improves precision to 5%, or better.

This could give a more-less objective performance gauge and comparison.
A control could be done by first running both websites (and tests) on one server - to confirm the test precision.
The downside is I have no idea how much RAM and CPU load do these test really put on a server. Probably ridiculously low, so the (oversold) server must be put to work by the rest of the stuff that's running on it.

Even more drivel, again - not an expert:
Running out of RAM should create problems, while (temporary) lack of CPU resources shouldn't hurt nearly as much. Think that's also something to be considered. At least with HDD and SSD used as swap RAM - not sure how NVME performs and if it can help change that, so that overloading CPU becomes the thing to worry about primarily. Sure experienced providers know the answers, while testing this takes time.

dahartigan

@AnthonySmith said:

@dahartigan said: You'd pretty much have to use alphawootracks as the benchmark for horrible performance imo lol

I think that was caused by them simply overselling rather than applying some intelligent design to it.

I agree with that 100% - One thing I just considered was that they also used HDD which was the cause of their main issues that I can recall (horrible IO) plus a lot of abusive clients that they attracted/appealed to/allowed who thrashed the CPU (admittedly, I'm guilty of that) and those who thrashed the network and disk (torrents etc).

I'm starting to get curious now about how far one could REALLY push a decent server, cram absolutely everyone onto that, oversell the absolute shit out of the CPU and RAM by just disregarding them basically lol. The storage would be interesting if oversold by no more than 2x (200% of physical storage) to see how much cramming would occur and how long that would take, and if monitored, could even be scaled by adding another drive or two.. I think I'm running off on a tangent from your idea, apologies for that, just brain farting because I have a spare 10 mins

poisson

Ok, let me bring some research and statistical expertise to the table.

First up, there are too many confounders (think of them as alternative explanations) that can influence perceptions of performance (which seems to be what we are measuring). I don't think it is possible to make any causal claims. The most we can say is that there is/isn't a statistically significant difference (meaning the difference is not due to chance) between both groups in terms of perception of server performance. That in itself is probably sufficiently valuable a conclusion.

To answer the question of sample size, well, that is a little complicated, but I would say at least 30 responses in each group would be a minimum.

sureiam

We can sit here and discuss the scientific faults or shortfalls of the setup all day long. Frankly this is also possible in even the best of scientific research. But this looks like a fun technical test and something uniquely possible to a forum like this. I'm sure enough members can find actual real world applications to run vs just testing to make this a worthwhile test (blogs, server monitoring, dns, discord, etc)

Let's do it and see what the results are. I have a feeling that a clean nvme raid 1 or raid 10 will actually have fairly acceptable use with very little performance impact. I'm personally curious as to what the typical number of users is for a low end provider. 100 users? 1000? We should maybe discuss that but the rest let's just see what happens!

InceptionHosting

Yep, the thing is with an artificial load and no way to tell how it will be used in advance and people also knowing it is a test there is no way to get concrete results.

It will however have a control group that will not know they are a control group. So some insight can be gained.

Additionally it will test an extreme resource density model that I have had in my head as an idea for a while that I am certain some hosts use but I have never had the balls or lack of morals to try in practice.

Also, it’s fun for the whole community.

poisson

@AnthonySmith said:
Yep, the thing is with an artificial load and no way to tell how it will be used in advance and people also knowing it is a test there is no way to get concrete results.

It will however have a control group that will not know they are a control group. So some insight can be gained.

Additionally it will test an extreme resource density model that I have had in my head as an idea for a while that I am certain some hosts use but I have never had the balls or lack of morals to try in practice.

Also, it’s fun for the whole community.

Frankly, there is no way to get generalizable results because in the first place, you will not get a probability sample for your participants. So, yes, this is a fun test and the results will be indicative but not predictive.

bikegremlin

@poisson said:

@AnthonySmith said:
Yep, the thing is with an artificial load and no way to tell how it will be used in advance and people also knowing it is a test there is no way to get concrete results.

It will however have a control group that will not know they are a control group. So some insight can be gained.

Additionally it will test an extreme resource density model that I have had in my head as an idea for a while that I am certain some hosts use but I have never had the balls or lack of morals to try in practice.

Also, it’s fun for the whole community.

Frankly, there is no way to get generalizable results because in the first place, you will not get a probability sample for your participants. So, yes, this is a fun test and the results will be indicative but not predictive.

Would it make sense to run a speed/stress test - was that intended?
For comparison of how performance changes depending on whether a server is oversold, or not.

InceptionHosting

@bikegremlin said:

@poisson said:

@AnthonySmith said:
Yep, the thing is with an artificial load and no way to tell how it will be used in advance and people also knowing it is a test there is no way to get concrete results.

It will however have a control group that will not know they are a control group. So some insight can be gained.

Additionally it will test an extreme resource density model that I have had in my head as an idea for a while that I am certain some hosts use but I have never had the balls or lack of morals to try in practice.

Also, it’s fun for the whole community.

Frankly, there is no way to get generalizable results because in the first place, you will not get a probability sample for your participants. So, yes, this is a fun test and the results will be indicative but not predictive.

Would it make sense to run a speed/stress test - was that intended?
For comparison of how performance changes depending on whether a server is oversold, or not.

Not really as that is fully synthetic.

The real point here is, can 250 experienced users coming to this without a destructive mindset:

A ) Tell that they are on a node which is running compressed ram as swap and secondary swap on NVME.
B ) Consider it usable regardless of if they can tell the difference.
C ) Somewhat outside of the end-user experience, it shows an interesting technical method of gaming significant density over standard and shows how well the model/method works.

That is all, no one is trying to get a paper peer-reviewed or published in a recognised journal, no one is saying there will not be areas of weakness in the method due to the scale, it is what it is and it is very interesting and maybe it poses more questions than is answers but that is not a bad thing either.

bikegremlin

I still don't see how a comparative performance test can harm anything. It would only give more info, that could be useful in judging performance reducing effect of overselling more objectively.

Without it, it's like avoiding to use a thermometer in comparing whether cellar is cooler than an air conditioned room. Sure - it's great to know the subjective impression. But why deliberately disregard any measurement?

InceptionHosting

@bikegremlin said: I still don't see how a comparative performance test can harm anything. It would only give more info, that could be useful in judging performance reducing effect of overselling more objectively.

Without it, it's like avoiding to use a thermometer in comparing whether cellar is cooler than an air conditioned room. Sure - it's great to know the subjective impression. But why deliberately disregard any measurement?

I have no issue with it in general but there is no doubt that the sold to spec server will perform better.

The test is a perception test at its core.

For example, we all know a v6 ford engine with a turbo is going to better than one without a turbo but if you want to find out if people think that the one without the turbo has an acceptable performance based on their past car experience as regular drivers of multiple vehicles then you don't put it on a rolling road to show performance metrics first as that will skew perception and perception is reality.

This is why I said from the getgo if this is done, and you buy it, if your only reason for buying is to find out which server is which then you missed the point.

Again though, I would expect to do the reveal at about 60 days in of a 90-day test so at that point comparative benchmarks could be a great addition to the test.

This reminds me of the tickets I get during black Friday like "I know its a black Friday special but I expect better performance". Upon checking the host node it becomes obvious that 30 customers are benchmarking at the same time with zero awareness of each other.

If 30 users benchmark on the undersold node it will get worse results than if no one is benchmarking on the oversold one

As such personally I put zero store by synthetic generic benchmarks on a VPS because you are doing it blind, so yes it does no harm if done at the appropriate time which is not at the start of the experiment and they would be better off being done by me at a time of known average load after 60 days has established that baseline.

If you want and are really interested I would be happy to have you keep that part honest by overseeing the method and execution.

InceptionHosting

@ReadyDedis said:
I'll be cool with sponsoring 2 units of 2 x Dual E5-2620v2 - 32GB RAM - 4x240GB SSD ( or 2x500GB SSD )

I am sorry, as the last comment on the page I missed it.

much appreciated, I would like to wait a few days more to see if we can get any NVMe based servers though as those pcie 3/4 lanes make a huge difference.