Can Promox VM network interface record foreign traffic

xyphos10 · January 2021

Hello all,

So I recently got a VPS from DataIdeas and after having set it up and placed zabbix agent on it I noticed something strange in the traffic graphs. Periodially like every 10 to 20 minutes the ens18 interface would show sustained download of about 100Mbps. That would last for about 10 minutes as well.

When I saw this I immediately logged into the VPS and started monitor the processes and traffic using htop, nload and tcptrack.

After waiting for about 20 minutes the sustained 100Mbps usage started again. Both nload and bmon showed ens18 downloading that much however htop showed no process using cpu and tcptrack showed that usage was 40KB/s.

The hosting provider using proxmox, does anyone know how else I can check what is downloading that much. Is it possible that the interface is recording traffic not from my vps?

ehab · January 2021

@yoursunny doing his pushups for sure.

# iftop

and check what is going on

xyphos10 · January 2021

@ehab said:
@yoursunny doing his pushups for sure.

# iftop

and check what is going on

Thank you, as was suspected it is some other user generating the traffic but getting recorded by my vps network interface. I guess it might have been UDP traffic which is why it was not shown in tcptrack, not entirely sure.

yoursunny · January 2021

@ehab said:
@yoursunny doing his pushups for sure.

I'm sleeping.

@xyphos10 said:
When I saw this I immediately logged into the VPS and started monitor the processes and traffic using htop, nload and tcptrack.

Yoy missed the most important tool: tcpdump.

sudo tcpdump -i ens18 -w 1.pcap

Wait 10 seconds, CTRL+C, transfer the file to your laptop, and open with Wireshark.

xyphos10 · January 2021

@yoursunny said:

@ehab said:
@yoursunny doing his pushups for sure.

I'm sleeping.

@xyphos10 said:
When I saw this I immediately logged into the VPS and started monitor the processes and traffic using htop, nload and tcptrack.

Yoy missed the most important tool: tcpdump.
sudo tcpdump -i ens18 -w 1.pcap
Wait 10 seconds, CTRL+C, transfer the file to your laptop, and open with Wireshark.

Noted for future troubleshooting, thanks.

Not_Oles · January 2021

Is it still happening? Any idea what it was and why it stopped? Thanks!

xyphos10 · January 2021

@Not_Oles said:
Is it still happening? Any idea what it was and why it stopped? Thanks!

I found the culprit of the traffic but I don't know why it is getting recorded in the network interface of my VPS and yes that person is still generating the traffic. I have opened a ticket with the provider about this situation.

Not_Oles · January 2021

I am trying to understand a little bit of what is happening. . . .

@xyphos10 said: getting recorded in the network interface of my VPS

If you have time, could you please explain, perhaps with a specific example, what "getting recorded in the network interface of my VPS" means? Also, how do you know that you "found the culprit?" Thanks!

Falzo · January 2021

this rather sounds like someone might be using your IP as gateway or has some strange routing/forwarding set up? or it's just broadcast traffic that is misdirected...

InceptionHosting · January 2021

It’s probably broadcast/multicast traffic.

ehab · January 2021

@AnthonySmith said:
It’s probably broadcast/multicast traffic.

do you know a way to using iptables or nft or magic to stop those. In one provider this traffic is measured in actual monthly BW.

InceptionHosting · January 2021

@ehab said:

@AnthonySmith said:
It’s probably broadcast/multicast traffic.

do you know a way to using iptables or nft or magic to stop those. In one provider this traffic is measured in actual monthly BW.

I mean you can at a host node level to some degree but it may impact legitimate use traffic as it is part of the functionality of networks generally, blocking it at a guest(VPS) level won't help because the traffic has to reach you in order to be blocked.

the correct way to block it is to give everyone their own /30 or maybe /29 so the broadcast domain is restricted I guess but the issue with that is on a per VPS level you get HUGE waste overall and obviously additional cost which would be passed on pretty much doubling prices in same cases.

I would suggest people simply open a ticket with hosts with data for example if you see a lot on port 17500 that usually means someone installed dropbox on their VPS and did not know to or were too ignorant to disable LAN Broadcasting, that I find is the biggest culprit, or at least used to be. there are probably tons of other things that do the same 90% of which is user-configurable, it should not be an issue for a host to pass on the request to the end-user generating the traffic.

In reality, though it is really a tiny amount of traffic overall in the wider scheme of things, it can be blocked/dropped with csf/iptables so that your server generates no action but obviously you want to avoid logging it as that will then have an even bigger impact

Falzo · January 2021

afaik misconfigured VPNs can also cause shitstorm of broadcast packages blown to wrong subnets...

xyphos10 · January 2021

@Not_Oles said:
I am trying to understand a little bit of what is happening. . . .

@xyphos10 said: getting recorded in the network interface of my VPS

If you have time, could you please explain, perhaps with a specific example, what "getting recorded in the network interface of my VPS" means? Also, how do you know that you "found the culprit?" Thanks!

So by getting recorded by the network interface I mean if you use tools like nload,bmon etc to view vps network usage it shows like my vps is the one causing that network usage.

I checked the control panel network usage graphs and it as well shows like if I am the one downloading for so long.

As suggested here I used iftop and noticed that it was traffic coming from ip x.x.x.2 causing this. Note my vps has ip x.x.x.229. As for what whoever owns vps with ip x.x.x.2 is doing to download so much data, the hostname is tor2-exit so you can guess.

What I am not too clear about is why the ens18 network interface of my vps is even showing like if this traffic is coming from itself

Here is a 12 hour zabhix graph of the network usage.

yoursunny · January 2021

Graph means nothing. Please capture and upload tcpdump traces in pcap or pcapng format, so that we can see exactly what's the traffic.

@AnthonySmith said:
give everyone their own /30 or maybe /29 so the broadcast domain is restricted

There's no need to allocate /30. You should use /31 that has the same effect but consumes half as many IP numbers.

xyphos10 · January 2021

Okay> @yoursunny said:

Graph means nothing. Please capture and upload tcpdump traces in pcap or pcapng format, so that we can see exactly what's the traffic.

@AnthonySmith said:
give everyone their own /30 or maybe /29 so the broadcast domain is restricted

There's no need to allocate /30. You should use /31 that has the same effect but consumes half as many IP numbers.

Finally got some free time, here is the packet capture https://transfer.sh/VWgBj/3.pcap

yoursunny · January 2021

@xyphos10 said:
@yoursunny said:

Graph means nothing. Please capture and upload tcpdump traces in pcap or pcapng format, so that we can see exactly what's the traffic.

Finally got some free time, here is the packet capture https://transfer.sh/VWgBj/3.pcap

I only looked at the first few packets. The bottom layers are all like this:

Frame 1: 2962 bytes on wire (23696 bits), 2962 bytes captured (23696 bits)
Ethernet II, Src: SuperMic_3d:3e:56 (3c:ec:ef:3d:3e:56), Dst: Dell_91:7d:7c (20:47:47:91:7d:7c)
Internet Protocol Version 4, Src: 173.249.5.76, Dst: 144.172.118.2

Notice that 144.172.118.2 is the destination IP address, not the source IP address.
Thus, it's not that this IP address is attacking you, but somehow the router/switch is directing the traffic toward this IP address into your container.

Some possible reasons:

There's a duplicate MAC address in the network. Both 144.172.118.2 and your box have the MAC address 20:47:47:91:7d:7c, so that both would receive the same frames.
The IP router thinks 144.172.118.2 has your MAC address 20:47:47:91:7d:7c. This could be caused by an incorrect static binding, or an ARP frame someone transmitted. However, an ARP frame should be valid for only 15 seconds, and I'm not seeing any ARP frames in the trace.
If your MAC address is not 20:47:47:91:7d:7c: the two ports are connected on an Ethernet repeater instead of the switch, so that all interfaces connected to the repeater would be seeing the same traffic.

One suspicious thing is that, you mentioned that your IP address ends with .229, but I'm not seeing any traffic toward that IP address. Moreover, you keep mentioning that this traffic is seen on ens18 interface.
Is there another interface on the system that you are using to access the box?
If you don't need the ens18 interface, you can just turn it off with sudo ip link set ens18 down command.

xyphos10 · January 2021

Hello, ens18 is the main interface of the vps

`

xyphos10 · January 2021

@yoursunny said: One suspicious thing is that, you mentioned that your IP address ends with .229, but I'm not seeing any traffic toward that IP address

I had filtered the tcpdump to only capture the x.x.x.2 traffic, here is a complete capture with no filtering https://transfer.sh/DXQ3v/5.pcap

yoursunny · January 2021

The screenshot indicates that your MAC address is not 20:47:47:91:7d:7c. You are not supposed to be receiving unicast traffic to that MAC address.
Something is messed up in the Ethernet switch.

The good news is, as long as your Ethernet interface is not in promiscuous mode (when you are not running tcpdump or other software that causes it to enter promiscuous mode), you would not be receiving such traffic.
It's possible that yje monitoring software that you used to collect the graph might be the culprit that triggers promiscuous mode.

InceptionHosting · January 2021

@yoursunny said: There's no need to allocate /30. You should use /31 that has the same effect but consumes half as many IP numbers.

There are other issues with going less than /30 by example broadcast does not work or rather you have no broadcast domain with a 31-bit mask and as such that may impact some functionality or cause some irregularity within some networks, it also still requires 2 IP's to be allocated.

I suppose /32's could be an option but that again comes with its own set of obstacles to overcome, a /32 is the smallest range you can be confident will be compliant in all practical use cases.

but really the bottom line here is, networks were designed at their core to have broadcast domains, the IP4 shortage has forced operation in shared IP space by independent users so it really is just something we need to live with, just my 2c

yoursunny · January 2021

@AnthonySmith said:

@yoursunny said: There's no need to allocate /30. You should use /31 that has the same effect but consumes half as many IP numbers.

There are other issues with going less than /30 by example broadcast does not work or rather you have no broadcast domain with a 31-bit mask and as such that may impact some functionality or cause some irregularity within some networks, it also still requires 2 IP's to be allocated.

/31 is defined in RFC3021.
It indeed doesn't allow directed broadcast, but broadcast doesn't make sense on a point to point link.
Sending to 255.255.255.255 is still allowed though.

/31 does require 2 addresses, but it's a 50% saving compared to /30.

I suppose /32's could be an option but that again comes with its own set of obstacles to overcome, a /32 is the smallest range you can be confident will be compliant in all practical use cases.

You cannot assign /32 on an Ethernet link because each end requires an address. However, it's possible to use /32 on PPP link.

but really the bottom line here is, networks were designed at their core to have broadcast domains, the IP4 shortage has forced operation in shared IP space by independent users so it really is just something we need to live with, just my 2c

The kernel understands /31 and PPP. PPP is widely used in residential networks, in the form of ADSL.
If someone's crappy application doesn't work, they should modify it to comply with RFC3021.

InceptionHosting · January 2021

Very idealistic, the bottom line is it is still using twice as many addresses and does not support everything as universally.

JBB · January 2021

Hey! I'm also seeing a crazy amount of traffic since Jan 8 as well. It looks like a ton of packets all destined for 144.172.118.2 (rDNS Tor2-Exit.AlienData.com). Not broadcast traffic, and not my MAC.
`
vnstat -d

ens18 / daily

     day         rx      |     tx      |    total    |   avg. rate
 ------------------------+-------------+-------------+---------------
  12/27/20    274.93 MiB |    3.09 MiB |  278.01 MiB |   26.99 kbit/s
  12/28/20    350.42 MiB |    3.20 MiB |  353.63 MiB |   34.33 kbit/s
  12/29/20    349.85 MiB |    3.07 MiB |  352.92 MiB |   34.26 kbit/s
  12/30/20    338.72 MiB |    1.16 MiB |  339.88 MiB |   33.00 kbit/s
  12/31/20    270.97 MiB |    1.36 MiB |  272.33 MiB |   26.44 kbit/s
  01/01/21    252.04 MiB |    0.98 MiB |  253.02 MiB |   24.57 kbit/s
  01/02/21    279.23 MiB |   19.17 MiB |  298.40 MiB |   28.97 kbit/s
  01/03/21    294.60 MiB |     620 KiB |  295.20 MiB |   28.66 kbit/s
  01/04/21    259.63 MiB |    7.17 MiB |  266.80 MiB |   25.90 kbit/s
  01/05/21    303.30 MiB |     575 KiB |  303.86 MiB |   29.50 kbit/s
  01/06/21    291.54 MiB |     710 KiB |  292.23 MiB |   28.37 kbit/s
  01/07/21    266.91 MiB |    4.47 MiB |  271.38 MiB |   26.35 kbit/s
  01/08/21    143.09 GiB |   18.51 MiB |  143.11 GiB |   14.23 Mbit/s
  01/09/21    182.14 GiB |    6.02 MiB |  182.15 GiB |   18.11 Mbit/s
  01/10/21    151.42 GiB |    3.19 MiB |  151.42 GiB |   15.05 Mbit/s
  01/11/21    167.92 GiB |    4.78 MiB |  167.92 GiB |   16.69 Mbit/s
  01/12/21    154.50 GiB |    4.48 MiB |  154.51 GiB |   15.36 Mbit/s
  01/13/21     39.93 GiB |  435.41 MiB |   40.36 GiB |   28.37 Mbit/s
 ------------------------+-------------+-------------+---------------
 estimated    283.25 GiB |    3.01 GiB |  286.26 GiB |

`

Actually, I was concerned before the 8th as well...this box is basically idle, yet lots of incoming traffic. When I investigated back in December, I was seeing a lot of DHCP requests from the router for other guests on the subnet. vnstat is estimating 2 TB of traffic for the month, so I'm going to open a ticket.

ehab · January 2021

@JBB said:

i have a similar problem , i will also open a ticket to understand what is happening, please update us when they say.

Can Promox VM network interface record foreign traffic

Comments