How to Become an One-Man NAT VPS Provider?

tang_cn · May 2023

Assume you have a dedicated server with /64 IPv6, and your goal is to create some IPv6 NAT VMs. How do you do it? Below is a tutorial about how to install QEMU/KVM and create VMs on Debian via the command line.

0. Advance Notice

Special thanks to @Not_Oles !
This tutorial has been verified to work with Debian 11 on Hetzner's dedicated server.
All commands are executed as root.
In this tutorial the host's network configuration is:
Network Interface: eno1
IPv4 Address: 192.0.2.2/26
IPv4 Gateway: 192.0.2.1
IPv6 Address: 2001:db8:ace:babe::1/64
IPv6 Gateway: fe80::1
We want to create a private network 192.168.0.0/24 for VMs; we want to use 192.168.0.1 as the IPv4 gateway and 2001:db8:ace:babe::1 as the IPv6 gateway for VMs.
We want to create a VM whose IPv4 address is 192.168.0.2 and IPv6 address is 2001:db8:ace:babe:cafe::1/80.

1. Enable IP Forwarding

Open /etc/sysctl.conf. Find net.ipv4.ip_forward; uncomment this line and set the value to 1. Do the same for net.ipv6.conf.all.forwarding.

Save the file; then run sysctl -p to apply changes.

2. Install Required Packages

apt install qemu-system qemu-utils libvirt-clients libvirt-daemon-system virtinst bridge-utils

3. Modify the Network Configuration

3.1 Make a copy of the original `/etc/network/interfaces`

cp -p /etc/network/interfaces /etc/network/interfaces.backup

3.2 Open `/etc/network/interfaces` and edit

Comment out eno1's IPv6 configuration, then add the following line:

iface eno1 inet6 manual

Continue to add the following lines to create an interface br0 for KVM networking:

auto br0
iface br0 inet static
    address 192.168.0.1/24
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
iface br0 inet6 static
    address 2001:db8:ace:babe::1/64
    up ip -6 route add default via fe80::1 dev eno1

bridge ports none means br0 is not attached to any physical interface. bridge_stp off disables Spanning Tree Protocol; usually we don't need it in simple cases. bridge_fd 0 sets the forwarding delay time to 0; 0 is good in simple cases. The iptables line allows LAN nodes with private IP addresses to communicate with external public networks. The ip -6 route line specifies the IPv6 gateway.

Save the file; then run systemctl restart networking.service to apply changes.

Example: my ORIGINAL /etc/network/interfaces

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eno1
iface eno1 inet static
    address 192.0.2.2
    netmask 255.255.255.192
    gateway 192.0.2.1
    # route 192.0.2.0/26 via 192.0.2.1
    up route add -net 192.0.2.0 netmask 255.255.255.192 gw 192.0.2.1 dev eno1
iface eno1 inet6 static
    address 2001:db8:ace:babe::1
    netmask 64
    gateway fe80::1

Example: my NEW /etc/network/interfaces

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eno1
iface eno1 inet static
    address 192.0.2.2
    netmask 255.255.255.192
    gateway 192.0.2.1
    # route 192.0.2.0/26 via 192.0.2.1
    up route add -net 192.0.2.0 netmask 255.255.255.192 gw 192.0.2.1 dev eno1
iface eno1 inet6 manual

auto br0
iface br0 inet static
    address 192.168.0.1/24
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
iface br0 inet6 static
    address 2001:db8:ace:babe::1/64
    up ip -6 route add default via fe80::1 dev eno1

4. Create the VM

virt-install --name YOUR_VM_NAME --ram MEMORY_SIZE_IN_MB --vcpus=NUMBER_OF_CORES --disk /PATH/TO/VIRTUAL/DISK/IMAGE.qcow2,device=disk,bus=virtio,size=DISK_SIZE_IN_GB,format=qcow2 --graphics vnc,listen=0.0.0.0,port=VNC_PORT,password=VNC_PASSWORD --network bridge=br0 --noautoconsole --cdrom /PATH/TO/ISOFILE.iso --boot cdrom,hd

Example

We want to create a VM whose name is TEST with 1GB memory, 1 core, and 10GB disk. TEST will be stored at /var/kvm/TEST.qcow2. The VNC port is 5901 and the VNC password is Hello1. We will use /var/iso/Debian.iso to install Debian on TEST.

virt-install --name TEST --ram 1024 --vcpus=1 --disk /var/kvm/TEST.qcow2,device=disk,bus=virtio,size=10,format=qcow2 --graphics vnc,listen=0.0.0.0,port=5901,password=Hello1 --network bridge=br0 --noautoconsole --cdrom /var/iso/Debian.iso --boot cdrom,hd

5. Configure Guest Networking

Because we haven't configured the DHCP service on the host, we have to manually set the VM network settings. If you are using a netinst image to install the OS, then you need to set the IP address to 192.168.0.2/24 and the gateway to 192.168.0.1 during the installation.

Example: Debain guest /etc/network/interfaces

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-hotplug ens3
iface ens3 inet static
    address 192.168.0.2/24
    gateway 192.168.0.1
iface ens3 inet6 static
    address 2001:db8:ace:babe:cafe::1/80
    dns-nameservers 2606:4700:4700::1001
    gateway 2001:db8:ace:babe::1

yoursunny · May 2023

IPv6 Address: 2001:db8:ace:babe::1/64

A big assumption is that the prefix should be routed toward the host machine.
This may not be the case at every dedicated server provider.

iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE

SNAT is more efficient when public IPv4 is static.

Because we haven't configured the DHCP service on the host, we have to manually set the VM network settings.

Nothing prevents your abusive customer to steal other's IP.
Moreover, port forwarding is missing.

somik · May 2023

Smart people doing what smart people does... Great tutorial @tang_cn

tang_cn · May 2023

@yoursunny said:
A big assumption is that the prefix should be routed toward the host machine.
This may not be the case at every dedicated server provider.

The idea is to create a bridge, so people should adjust it based on their network settings.

SNAT is more efficient when public IPv4 is static.

Thanks for reminding me SNAT, I should change it. Because the performance difference between MASQUERADE and SNAT is too small, I usually use MASQUERADE. People who are interested in SNAT can try it.

Nothing prevents your abusive customer to steal other's IP.
Moreover, port forwarding is missing.

This is a very basic tutorial, and I want to make it as simple as possible, that's why I didn't include DHCP. If someone really tries to sell NAT VPS, there are many more things to consider aside of DHCP (e.g., disable certain CPU flags, expose CPU info directly to the guest, PCI passthrough, etc)

yoursunny · May 2023

If someone really tries to sell NAT VPS

Your title says this is for NAT providers, hence these need to be included.

tang_cn · May 2023

@yoursunny said:
Your title says this is for NAT providers, hence these need to be included.

Cause I just want to add humor to the title.

A comprehensive tutorial for providers should also include how to setup control panels, how to file tax, how to resolve a DMCA Complaint, etc. Apparently this isn't my goal

Oh, note my title is ending with a question mark, so I'm asking a question in the title; my title is not "Tutorial for NAT VPS providers."

FrankZ · May 2023

@tang_cn - I definitely understood that the title was tongue in cheek humor after opening the thread, although given the vast variety of members here at LES it is unlikely that everyone will take it that way. I appreciate the time you took to write this tutorial for those of us that want to set up NAT VMs as a hobby or as a learning experience.
(If you want to change the title in the future please DM me.)

@yoursunny - You have much more knowledge and experience setting up and working with NAT VMs than most of us here at LES. I am sure that you can add much positive information to this tutorial and I hope you will do so.

Hopefully we can get more tutorials written by members here at LES if other members can add information to the OP with out making the member starting the thread feel like they are being criticized for writing something they thought would be helpful. Very few people know everything about a subject, but collectively LES members seem to know a lot about many things. Hopefully we can start a new trend of working together to write some great tutorials and reference guides that help many old and new members alike.

@somik said: If possible, may I request a separate category for "Tutorials" so that it wont get mixed up in other technical discussions and such?

Previously tutorials were done under the "LES Talk" category and I have moved this thread there for now. I will talk with @Mason and see if he wants to make a separate "Tutorials" category or would prefer that they continue to be put in "LES Talk", as this is above my pay grade.

somik · May 2023

@FrankZ said:
@tang_cn - I definitely understood that the title was tongue in cheek humor after opening the thread, although given the vast variety of members here at LES it is unlikely that everyone will take it that way. I appreciate the time you took to write this tutorial for those of us that want to set up NAT VMs as a hobby or as a learning experience.
(If you want to change the title in the future please DM me.)

@yoursunny - You have much more knowledge and experience setting up and working with NAT VMs than most of us here at LES. I am sure that you can add much positive information to this tutorial and I hope you will do so.

Hopefully we can get more tutorials written by members here at LES if other members can add information to the OP with out making the member starting the thread feel like they are being criticized for writing something they thought would be helpful. Very few people know everything about a subject, but collectively LES members seem to know a lot about many things. Hopefully we can start a new trend of working together to write some great tutorials and reference guides that help many old and new members alike.

If possible, may I request a separate category for "Tutorials" so that it wont get mixed up in other technical discussions and such?

@somik said: If possible, may I request a separate category for "Tutorials" so that it wont get mixed up in other technical discussions and such?

Previously tutorials were done under the "LES Talk" category and I have moved this thread there for now. I will talk with @Mason and see if he wants to make a separate "Tutorials" category or would prefer that they continue to be put in "LES Talk", as this is above my pay grade.

Thanks! Hope it gets approved!!!

Not_Oles · May 2023

Thanks to @tang_cn for this great tutorial!

I've just enjoyed investing the past hour or two looking through the tutorial and also looking at various files and directories on the server. For me, this is the first time I glanced at libvirt. So far, I mostly have tried to focus on qemu, because it seemed to make sense to start with qemu rather than with an additional mechanism which runs on top of qemu. I'm certainly not trying to say anything against libvirt or against anybody who uses libvirt. But it has seemed possible to begin with very simple configurations and calling qemu directly.

Special thanks to @yoursunny for his unfailingly insightful, generous, kind, and gentle comments. Although port forwarding didn't seem to make it into the tutorial, port forwarding was not omitted by @tang_cn. Port forwarding was handled by a separate script called, surprisingly, port_forward.sh. Since it's not yet clear to me why port forwarding was handled separately, I will let @tang_cn comment further, if he wishes.

In case anybody might be interested, this, according to ps aux, was the command that libvirt sent to qemu to start a VPS:

libvirt+    1042  1.2  0.2 3070960 371624 ?      Sl   03:06   0:08 /usr/bin/qemu-system-x86_64 -name guest=tang_cn,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-tang_cn/master-key.aes -machine pc-i440fx-5.2,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram -cpu Skylake-Client-IBRS,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off -m 2048 -object memory-backend-ram,id=pc.ram,size=2147483648 -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid redacted -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -blockdev {"driver":"file","filename":"/var/kvm/redacted.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null} -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2 -device ide-cd,bus=ide.0,unit=0,id=ide0-0-0,bootindex=1 -netdev tap,fd=34,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=redacted,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:redacted,password -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

I am extremely pleased that @tang_cn used a MetalVPS server to test his command line libvirt VM! I am delighted to see @tang_cn's tutorial and similar wonderful content appearing so often here at LES! Special thanks to @FrankZ for his lively insights into humor, motivation, and the benefits of kind co-operation.