Connection being bottle necked by lack of resources.

tony · May 2020

Since a few days I experienced instability on my VPS, it may be since the beginning but I really saw it when my VPN started to work. I want to share my measurement techniques to know if you think they are reliable enough and if some of you experienced the same kind of problems. The support answered to me that Hetzner may be under a DDoS atack.

On the VPS I run :

iperf -s -p 19003

On my Linux box (wired) :

iperf -c vps -p 19003 -i 0.5 -t 300 >> bw.log

I then rework the log file with cut, sed and a little bit of manual editing to put the data in Excell and generates the graphs attached.

I also made the measurements using ipv6 and the results are better, not perfect but at least it does not fall to zero for long periods.

InceptionHosting · May 2020

how much Ram do you have?

tony · May 2020

@AnthonySmith said:
how much Ram do you have?

256MB
Edit: after some more tests it looks like it is more a CPU throttling or something like that which causes the network freeze. I continue the investigation.

InceptionHosting · May 2020

@tony said: 256MB
Edit: after some more tests it looks like it is more a CPU throttling or something like that which causes the network freeze. I continue the investigation.

You have a VPS with 1998 RAM in a super high contention environment while testing it with 2020 tools, I would not put much store by your results tbh.

If it works for what you need great, if it does not, pay more, you are at the super ultra mega budget end of the spectrum.

You don't enter a skateboard in to a NASCAR race then complain about the 0-60 acceleration

tony · May 2020

I understand that this is low end but I also like to understand what happens. Also I will be the only user of this service and I can deal with less performances but the periodic freezes are definitely problematic.

I can see server side the OpenVPN process rising to 100% CPU and then everything freezes. I am currently testing different Cipher to see if I can improve the performances, I may also try to throttle bandwidth or limit simultaneous connexions.

InceptionHosting · May 2020

I have gone ahead and updated the title to reflect reality rather than give bad PR then.

schnafi · May 2020

I had the same problem on another machine with 256mb ram I am using. Even ssh was not usable and the system hung every couple of seconds. I fixed it by adding 512MB swap. Don't know why it happened, but it fixed the server freezes permanently.

InceptionHosting · May 2020

@schnafi said: Don't know why it happened, but it fixed the server freezes permanently.

It happened because you were probably trying to squeeze so much in that there was barley any room for anything else, 256mb ram on kvm/xen is significantly less than is required to run unless you are prepared to do some expert level tuning.

So you essentially downloaded more ram lol.

You cant add swap on containers/vz

tony · May 2020

@schnafi said:
I had the same problem on another machine with 256mb ram I am using. Even ssh was not usable and the system hung every couple of seconds. I fixed it by adding 512MB swap. Don't know why it happened, but it fixed the server freezes permanently.

I already have 512MB of swap and even with very low resource intensive tasks like iperf (low on CPU and RAM, not on network) i get freezes. For instance this is the output of iperf :

[  3] 23.0-23.5 sec  7.88 MBytes   132 Mbits/sec
[  3] 23.5-24.0 sec  5.38 MBytes  90.2 Mbits/sec
[  3] 24.0-24.5 sec  5.25 MBytes  88.1 Mbits/sec
[  3] 24.5-25.0 sec  5.14 MBytes  86.3 Mbits/sec
[  3] 25.0-25.5 sec   764 KBytes  12.5 Mbits/sec
[  3] 25.5-26.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 26.0-26.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 26.5-27.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 27.0-27.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 27.5-28.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 28.0-28.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 28.5-29.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 29.0-29.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 29.5-30.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 30.0-30.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 30.5-31.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 31.0-31.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 31.5-32.0 sec  0.00 Bytes  0.00 bits/sec>

Why does it freezes even if the CPU is used at less than 20% and RAM is at 24MB used of the 512MB and swap is empty ?

I have the feeling that something else on the Server (outside of my container) is trying to get resources and then some containers are freezed.

tony · May 2020

By limiting the bandwidth to 10Mb I get much more reliable performances and almost no freeze but there is still (fewer and shorter) freezes.

$ iperf -c vps -p 19003 -i 0.5 -t 300 -b 10M 
[  3] 75.0-75.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 75.5-76.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 76.0-76.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 76.5-77.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 77.0-77.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 77.5-78.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 78.0-78.5 sec   768 KBytes  12.6 Mbits/sec
[  3] 78.5-79.0 sec   512 KBytes  8.39 Mbits/sec
[  3] 79.0-79.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 79.5-80.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 80.0-80.5 sec   768 KBytes  12.6 Mbits/sec
[  3] 80.5-81.0 sec  28.2 KBytes   462 Kbits/sec
[  3] 81.0-81.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 81.5-82.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 82.0-82.5 sec  0.00 Bytes  0.00 bits/sec
[  3] 82.5-83.0 sec  63.6 KBytes  1.04 Mbits/sec
[  3] 83.0-83.5 sec  1.25 MBytes  21.0 Mbits/sec
[  3] 83.5-84.0 sec  3.00 MBytes  50.3 Mbits/sec
[  3] 84.0-84.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 84.5-85.0 sec   640 KBytes  10.5 Mbits/sec
[  3] 85.0-85.5 sec   640 KBytes  10.5 Mbits/sec
[  3] 85.5-86.0 sec   640 KBytes  10.5 Mbits/sec

At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

WSS · May 2020

As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

InceptionHosting · May 2020

@tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

tony · May 2020

@WSS said:
As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

@AnthonySmith said:

@tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable

InceptionHosting · May 2020

@tony said:

@WSS said:
As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

@AnthonySmith said:

@tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable

what do you get when not trying artificial benchmarks e.g. :

wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

vimalware · May 2020

I doubt you need more than 7-8 threads. 6 should cover almost every provider I have.

tony · May 2020

@AnthonySmith said:

@tony said:

@WSS said:
As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

@AnthonySmith said:

@tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable

what do you get when not trying artificial benchmarks e.g. :

wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

InceptionHosting · May 2020

@tony said:

@AnthonySmith said:

@tony said:

@WSS said:
As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

@AnthonySmith said:

@tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable

what do you get when not trying artificial benchmarks e.g. :

wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

It is more than enough, I am trying to get you to test and post the results of that because it simulates real world use, in what world are you downloading a 10GB file on to a VPS that cost less than your lunch?

Honestly if I had to guess based on your comments so far i would assume you have been IO de-prioritized due to the sorts of tests you are running.

tony · May 2020

@AnthonySmith said:

@tony said:

@AnthonySmith said:

@tony said:

@WSS said:
As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

@AnthonySmith said:

@tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable

what do you get when not trying artificial benchmarks e.g. :

wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

It is more than enough, I am trying to get you to test and post the results of that because it simulates real world use, in what world are you downloading a 10GB file on to a VPS that cost less than your lunch?

Honestly if I had to guess based on your comments so far i would assume you have been IO de-prioritized due to the sorts of tests you are running.

I don't understand what you are trying to prove. The problem was here way before I started the tests. If doing apt dist-upgrade made me IO de-prioritized then the service is not cheap anymore regarding what is provided

schnafi · May 2020

@AnthonySmith said:
You cant add swap on containers/vz

Interesting, I did not know that. Another (new) limitation of OVZ. Regardless, I still prefer OVZ over KVM on low ram machines, where only a minimum of services are running, as they are mostly cheaper.

tony · May 2020

@tony said:

@AnthonySmith said:

@tony said:

@AnthonySmith said:

@tony said:

@WSS said:
As Anthony said, you're not going to get great bandwidth with low RAM; as directly demonstrated above with your attempt to run 18 threads. iperf uses a hell of a lot of bandwidth to test, which also requires RAM. Your resources are exhausted. Try using with -n 1 to see how well you manage with one thread. I bet it's a lot higher for that thread than even the 4 available.

Get a service which fits your needs, but don't blame the provider when you get the bargain-basement service.

The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

@AnthonySmith said:

@tony said: At this level the CPU do not even reach 1% (with htop running) and le load is under 0.2 so again, I understand that this is low end but there is still something wrong somewhere ?

The ram still needs to buffer everything, when you run out of ram even though that may not show up as used ram in htop between refreshes it then swaps around which is disk based, that's just how things work, also htop itself uses more than 1% cpu usually to run so I am not sure how reliable the data you are looking at is.

Probably better you install atop so you can get a lot more detail, in any event get more ram the issue will go away.

I guess it should be able to deal 10Mb (1.2MB !) per second. I already had two VPS before, also based on OpenVZ, I think one of them was 128MB of ram and I never experienced this kind of freezes. SSH session are just unusable

what do you get when not trying artificial benchmarks e.g. :

wget http://speedtest.ams01.softlayer.com/downloads/test100.zip

100MB file is not enough, so I took the freedom to use this one : https://speed.hetzner.de/10GB.bin. After a few seconds the download freeze. I am looking for a way to get a non interactive output from wget or curl.

It is more than enough, I am trying to get you to test and post the results of that because it simulates real world use, in what world are you downloading a 10GB file on to a VPS that cost less than your lunch?

Honestly if I had to guess based on your comments so far i would assume you have been IO de-prioritized due to the sorts of tests you are running.

I don't understand what you are trying to prove. The problem was here way before I started the tests. If doing apt dist-upgrade made me IO de-prioritized then the service is not cheap anymore regarding what is provided

There is also the 500GB limit per month to prevent abuse.

InceptionHosting · May 2020

@tony said: I don't understand what you are trying to prove.

OK I will opt out of this discussion then.

Hetzner_OL · May 2020

@tony - If you think there is a problem on Hetzner's end, please write back to our support team using the same support ticket as you mentioned at the beginning of this thread. Explain why you think the issue is on Hetzner's end. Provide our team with some details of your use case and what troubleshooting you have done. Perhaps they will give you similar feedback to what you have seen here or maybe they will have other insights. -Katie

tony · May 2020

@Hetzner_OL said:
@tony - If you think there is a problem on Hetzner's end, please write back to our support team using the same support ticket as you mentioned at the beginning of this thread. Explain why you think the issue is on Hetzner's end. Provide our team with some details of your use case and what troubleshooting you have done. Perhaps they will give you similar feedback to what you have seen here or maybe they will have other insights. -Katie

Thank you, I am also discussing with the customer support trying to catch the problem. As I know these offers are supposed to be support-less I don't wan't do be too invasive regarding support .

WSS · May 2020

@tony said:
The -n option is for the number of bytes to transmit, nothing related to the number of threads, by default it seems to use 3 threads server side, th 18 lines you see are not parallel, they are sequencial.

You are correct. I meant -P for parallel. I am far too use to -n being number of processes.

@tony said:
Thank you, I am also discussing with the customer support trying to catch the problem. As I know these offers are supposed to be support-less I don't wan't do be too invasive regarding support .

So, instead you'll shit all over them on the forum trying to find a specific way to get 1000jiggabits on a 256MB plan? Did you get kicked off of Virmach already?

If you didn't tag me like 8 fucking times in every response due to bandwidth waste, I would have just ignored you.

tony · May 2020

Sorry for tagging you untimely, it was not my intention, I just wanted to quote what I was answering for clarity. Also I did not shit on anybody, if I identified the provider, its because I think that it is a useful information to identify the problem. I am not trying to get thousands of Gb continuously, sorry if the benchmark to highlight the problem was not appropriate but even 1Mb for 2 minutes is problematic ... Finally I did not get ban from anywhere as far as I remember, if the "tacit" rules of low end VPS changed since I used one a few month ago, then fine, I am just trying to understand.

WSS · May 2020

You can likely just take care of this by adding more RAM. If they're looking from the node and there's no issue on the node, the bottleneck is obviously your instance. I'm not sure of ANYTHING being happy in 256MB amymore - I use almost that with my TinyKVM nameservers with no other external-facing services on them.

mikho · May 2020

It’s always fun trying to measure speed/connectivity during a ddos attack.

Reading this thread I assume I’m the provider as there are som signs.

Any who, one of my DE Nodes has been under daily DDoS attacks for roughly a week now.

Hetzner is doing a great job keeping the node online but its not as stable as it used to be.

Hope it will end soon.

tony · May 2020

@mikho said:
It’s always fun trying to measure speed/connectivity during a ddos attack.

Reading this thread I assume I’m the provider as there are som signs.

Any who, one of my DE Nodes has been under daily DDoS attacks for roughly a week now.

Hetzner is doing a great job keeping the node online but its not as stable as it used to be.

Hope it will end soon.

No problem, I guess the timing was unfortunate as I ordered the VPS at the beginning of the DDoS attack. Please let me know when it is over so I can see if the situation effectively improved.

Connection being bottle necked by lack of resources.

Comments