logging outgoing connection IP with tcpdump / any alternative
Hi,
I am receiving a lot of abuse reports at my sg location, I am here seeking advice(read: help) from the LES Scholars committee.
short:
some VPS Containers are assigned internal ip's with a NAT'ed connection to the outside IPv4 world via a single shared public IPv4.
I suspect there are 1-2 sick people who are abusing this shared connection, attacking, attempting to ssh brute-force & port-scan other servers.
So the public ip4 gets listed on blacklists & I receive reports containing affected server details, IP etc,
I'm required to take corrective actions & logging/analysing connections from internal IPs...need some help!
Currently i use tcpdump to log connections & trace the internal origin IP of attack.
tcpdump -n -i venet0 -G 86400 -w %F.cap
daily log size reaches as large as 22GB...
Any changes to tcpdump I could do? a way to reduce the log size..
I only need to log outgoing connections over IPv4, logging the date, source IP & destination IP.
has someone else faced similar situation?....alternatives to tcpdump?...should I use tshark?(how?)
any modifications I can do with the tcpdump command ?
any & all suggestions are very much appreciated.
Thanks for taking your time reading this/helping.
Comments
@AnthonySmith is your ultimate advise.
tcpdump -vv -nNxXi venet0 src net 192.168.0.0/24 -s 0 proto TCP and port 22
obviously, replace 192.168.0.0 with your local range, that would be what I would use to watch it, probably with some | grep to only get the headers, in terms of logging it, sorry, there is no simple way that wont take up a large amount of space.
I suppose you could be clever and try something like:
netstat -anp | grep :22 | grep ESTABLISHED | wc -l >/tmp/ssh-session-count
Then write a script that checks the number in /tmp/ssh-session-count and logs for 5 minutes then emails you an alert I am about to take the dog out right now so cant knock one up for you but it is not a hard script.
tbh I have seen a big rise in NAT boxes getting rooted in the past few weeks and attached to botnets it is almost ALWAYS caused because people sign up with password123 or rootpass or zaq1xsw2 etc.
It is usually a process pretending to be rsync running as 'tsm' from a tmp or 'hidden' directory like /usr/var/.x25/
try
ps aux | grep tsm
almost all the ones I have discovered that generated abuse reports form ssh brute-forcing or scanning have been running as that.I assume there is someone specifically targetting NAT ranges that has become aware of them because they are on a single IP and all use the same port format and so many people think they are safe behind NAT or they think "done with this VPS I will reinstall and leave it" not realising they are leaving it with their dumb initial root password exposed.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
Thanks for writing this, gives me a basic idea of how other people are doing it.
Can use something like awk print for getting the headers (I think)
The script part looks promising, will try writing one & if successful share here incase someone needs.
I specifically appreciate that you helped, knowing I do a similar product & may be considered a competitor.
https://webhorizon.net
That has never been a factor in any decision I make, I am just not wired that way.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
Shit yea, I discovered one user 2 days ago with tsm cuz using CPU like anything, a load average of 4 on a 256MB NAT
I did suspend him yesterday, so I guess it's over...I wonder if the user is involved or is it they have no idea what is going on with their VPS...
https://webhorizon.net
Most likely the latter...
dnscry.pt - Public DNSCrypt resolvers hosted by LowEnd providers • Need a free NAT LXC? -> https://microlxc.net/
"never attribute to malice that which is adequately explained by stupidity"
Hanlons razor!
https://webhorizon.net
Well, NAT is a bitch yea.
Due to the recent events, I started writing a python script, that does that for me.
Basically all what it does is, parsing a json file and checks if anything suspicious has rizen.
If so, a notification will be send to us with information + a tcpdump will be started.
I am not the fan, of tcpdump traffic randomly, because there may be something.
I do not wanna share it pubic, because reasons, I tried to make it as dynamic as possible, so minimal hardocded values.
But I may share it with NAT Providers.
Free NAT KVM | Free NAT LXC | Bobr
ITS WEDNESDAY MY DUDES
The
-w
flag stores packet header and payload, so the output is huge.Use
-s
flag to reduce snaplen, so that you only save the IPv4 header.tcpdump has no concept of connection, and cannot identify outgoing vs incoming connection. However, you can add eBPF filter to capture outgoing packets only.
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
Wiretapping all customer traffic is something you really want to avoid...
Besides reducing snaplen, you also only care about SYN packets (ie, packets with SYN flag set and RST,ACK,FIN flags clear). You can do that with an appropriate tcpdump filter parameter. But it should also be possible to use iptables rules to count SYN packets per IP.
@aeg Thanks, ofc not wiretapping traffic only the source & destination IPs for the mandatory requirements & finding the abusers...only syn packets. prefer keeping iptables untouched & using tcpdump.
https://webhorizon.net
Well, tcpdump is the old way, People have thought about this already and created pmacct, which is an network accouting deamon.
http://www.pmacct.net/
Pretty nice, since @FHR reffered me to it, I use it for all sort of stuff.
Just slap some monitoring on top, with a notification gateway like https://gotify.net/ = profit
I called our creation "GateKeeper", it does pretty well.
The next time, we get an Abuse, we can just look into your notifications and nail that abuser down.
Free NAT KVM | Free NAT LXC | Bobr
ITS WEDNESDAY MY DUDES
Nice name ; I looked into the project page.
So now won't have to spend almost 2 hours investigating an abuse case.
https://webhorizon.net
I'm surprised that everyone is suggesting a completely incorrect approach.
You don't want to do full packet capture (aka tcpdump), you want only metadata capture.
Look into hsflowd (saves traffic and sends it as sFlow) and sfcapd (listens for sFlow and saves it into a file). You can then use sfdump for analysis.
This is the approach pretty much every ISP uses for bulk traffic logging.
It's also possible to use pmacct as @Neoon suggested, however I don't think it's a good fit for this purpose.
SkylonHost.com High Bandwidth European Cloud KVM | AS202297
@FHR I suspect that will not work on a venet0 interface with NAT containers, I will check it out though.
https://inceptionhosting.com
Please do not use the PM system here for Inception Hosting support issues.
There are several ways with which hsflowd can sample the traffic. I like using IPTables (nflog) to redirect traffic to it.
https://blog.fhrnet.eu/2019/10/08/retaking-control-of-your-network-part-1/
Basically you should be able to place the rules in before the NAT takes place.
SkylonHost.com High Bandwidth European Cloud KVM | AS202297
Works fine for this purpose, better than expected.
However no idea how it does perform on 10gig+ nic's, performance wise.
Free NAT KVM | Free NAT LXC | Bobr
ITS WEDNESDAY MY DUDES
pmacct looks interesting! It's GNU GPL v2!
I hope everyone gets the servers they want!