Turns out I've been working (for pay) with computers for over 30 years now, and I've tried and learned a lot during that time (still learning, it's a dynamic field). Here's a short-list of what's worked for me so far (in an infrastructure-related order so to speak):

For those who know my style - the series is mostly technical, brief, and to-the-point. That's what it is: basically a list with brief explanations why. Not my usual long-windedness.

My Hosting stack
Infrastructure - domain registration, DNS, website and email hosting.

My (backups) Storage stack
Storage - physical and cloud.

My OS & System Software stack
Computer-running - OS, drivers, security, diagnostics, and utilities.

My Application Software stack
Computer-using - writing, editing, playing media, and troubleshooting.

Writing this series got me to revisit and rethink my stack (helping myself too), and took me on a few trips down memory lane (like first time seeing Total Commander on my late uncle's PC, while I was just learning and using DOS command prompt at the time and similar).

It was fun, and, as the modern managers and HR folks say: challenging. LOL

Relja

P.S.
I think this topic logically belongs in this section, but if it's a bad idea, moving is one-click away.

I think I want a really peaceful, retro green terminal like the above screenshot.

I recently installed and wrote about vscodium and code-server, but vscodium, although super great, is not retro green.

In the retro green terminal, there is a peaceful, non-distracting, empty space into which I can load whatever I am working on at the moment. Any one of several editors plus a vast array of command line tools can be used instantly.

In addition to the command line tools, dot files and dot directories can be used keep everything arranged, close by, yet out of the way. Right now I use one dot file and two dot directories.

My dot file is .tlog, which is a simple text file with dates, log entries, comments, and terminal commands transcribed along with their output.

My two dot directories are .current and .attic. The first, .current, is where I keep individual files and project directories on which I currently am working, although not at the precise present moment. The second, .attic, is where I keep files and projects on which I am not currently working, but which I think I might need or want to see again before too long.

Once a new project hits .current or moves from .current to .attic, I still can go work on it there. Its not like I have to move the project back to my home directory in order to edit a file.

As a simple workflow example, here is me working on this post, comments added.

root@tx:~# su - tom # Logging in as root saves me from typing sudo a million times as day. 
tom@tx:~$ date # I need to remind myself of the date in order to add the date to the log.
Mon Oct 27 05:16:31 PM UTC 2025
tom@tx:~$ vi .tlog # Starting todays log entries. 
tom@tx:~$ cp -p .tlog .tlog~ # Local quick backup, just in case.
tom@tx:~$ vi How-do-you-set-up-your-terminal # Write todays post.j 
tom@tx:~$ ls # Does the post exist in my home directory?
How-do-you-set-up-your-terminal # Yes!
tom@tx:~$ vi .tlog # Add log entry for this post.
tom@tx:~$ ls .current/ # Not much yet on the new Texas server.
tom@tx:~$ mv How-do-you-set-up-your-terminal .current/ # Move the draft to the .current directory.
tom@tx:~$ clear # Restore the peacefulness of the clean view of my home directory.

The green color I am using is #00ED00, which reminds me of the IBM 5151 that I used to use.

For backup, I mirror my /home/tom directory on another server from a different provider. I can log in to either mirror and begin working right away. In addition to mirroring, I keep tar backups at a third provider and also on a local disk.

The biggest disadvantage of my retro green setup might be that it is stuck in time. I loved and lost the infinite scrollback from the slightly earlier days of paper terminals. Also, perhaps my love of retro green keeps me from embracing many advantages of newer terminal setups?

How do you set up your terminal? Can you convince me, almost 78 years old now, to move forward in time, to become young again?

code-server provides the ability to "[r]un VS Code on any machine anywhere and access it in the browser."

The vscodium README sends us to the @paulcarroty repo at Gitlab where there are specific Debian apt install instructions.

Install vscodium and/or vscodium-insiders

The specific install commands can be copied directly from the @paulcarroty repo. Basically there are only four simple steps:

• Install a gpg key;

• add the repository to /etc/apt/sources.list.d/;

• update, upgrade; and then

• install codium and/or codium-insiders.

Install code-server

Next is to install code-server. The code-server repo includes an install script, as well as the curl commands to test and run the install script.

Connecting

To connect, set up an ssh tunnel to the VPS from the local machine. Then, on the VPS, start codium or codium-insiders followed by code-server. Finally, go to http://localhost:8080 in the local machine's browser.

On the local machine:

• ssh tom@xxx.xxx.xxx.xxx -L 8080:localhost:8080

On the VPS:

• $ cat /home/tom/.config/code-server/config.yaml # Get the code-server password

• $ codium # Start vscodium.

• $ code-server # Start code-server.

Back on the local machine:

• Go to http://localhost:8080 in the browser.

Screenshots

The new FOSSBilling free and open source billing platform was discussed here at LES in December 2022. The 2022 discussion included very helpful background posts from FOSSBilling developers @jaapmarcus and @evrifaessa.

FOSSBilling has progressed a lot since the December 2022 LES discussion. Indeed, shared hosting providers already have been using FOSSBilling in production.

Recently, I signed up as a customer at two shared hosting providers who use FOSSBilling. This post details the results of my testing FOSSBilling from the perspective of a shared hosting customer. Also presented after the test results are, courtesy of @BelleNottelling, insights about the upcoming 0.6.0 FOSSBilling release along with tips for using FOSSBilling.

FOSSBilling Demo

The FOSSBilling website has a great demo, which enables everyone to see both the administrator panel interface and the client area interface.

FOSSBilling vs. Blesta

When FreeVPS.org was started, the FreeVPS Team went with Blesta instead of FOSSBilling because the FreeVPS Team had experience with Blesta. However, unless, like FreeVPS, you have a generous donor of a Blesta license (Thanks @Cloudie!), using Blesta normally requires monthly payments or more expensive lifetime licenses. Also, Blesta is not 100% open source.

FOSSBilling Tested At HostByBelle And At YetiNode

Shared hosting providers already using FOSSBilling in production include HostByBelle and YetiNode. Belle is one of the FOSSBilling developers. She is @BelleNottelling here on LES. YetiNode is run by our very own indomitable Yeti, @AuroraZero.

My overall experience testing FOSSBilling as a customer was excellent at both HostByBelle and at YetiNode. I had no trouble launching a test html page on each provider's shared hosting service. Here are links to the two test pages: HostByBelle and YetiNode.

Both providers use HestiaCP to provision shared hosting. Billing with FOSSBilling and provisioning with HestiaCP were very similar at HostByBelle and at YetiNode. I tried FOSSBilling's Support Ticket interface at HostByBelle. @AuroraZero talked with me by PM here at LES.

Screenshots Of FOSSBilling At HostByBelle

The gorgeous dark theme shown in these screenshots comes from the reverse colors engineered by Chrome Browser's High Contrast Accessibility Extension. These screenshots are what I actually saw during sign-up at HostByBelle. If you also wish to see the normal, non-reversed color mode, please look at the above linked FOSSBilling demo.

• Sign-Up

• Dashboard

• Ordering

• Payment

• Hestia Control Panel

• Invoices

• Support Tickets

• Quick Ticket Reply

Additional FOSSBilling Discussion

• Emails

FOSSBilling sent transactional emails every step of the way, including sign-up, payment, and support tickets. The email setups at HostByBelle and at YetiNode differ slightly. HostByBelle uses a transactional email provider, and YetiNode self-hosts its email MTA.

• Payment Providers

FOSSBilling at HostByBelle offered Paypal and payment from account credit. Payment wasn't involved with YetiNode, because I tested their free tier instead of their paid tier. Paypal payment worked fine at HostByBelle.

• Provisioning Shared Hosting and VPSes

Like most billing panels, FOSSBilling requires the use of another additional software package to provision hosting services. Sometimes, modules can be added to billing panels to send instructions to the additional provisioning software's API.

As shown at both HostByBelle and at YetiNode, FOSSBilling works well with HestiaCP for shared hosting. FOSSBilling's five (including Hestia) already-working provisioning integrations are listed in the FOSSBilling Documentation. Integration of Proxmox also has been discussed on the FOSSBilling Forum and, more recently (8/21/2023), on the FOSSBilling Discord.

• FOSSBilling Forum

If you have additional questions about FOSSBilling, you might want to check out the FOSSBilling Forum.

HestiaCP

• App Installation

As noted above, both HostByBelle and YetiNode use Hestia Control Panel for shared hosting installs. Here is HestiaCP's One Click App Installation Panel as seen at YetiNode.com. As expected, the HestiaCP app installation is similar at HostByBelle.

• Let's Encrypt Certificates

SSL certificates from Let's Encrypt were provisioned with one click at both HostByBelle and YetiNode.

FOSSBilling Updates

Both FOSSBilling's client theme and its multiple administrative user permissions hopefully will be updated in the next 0.6.0 release or soon thereafter. Belle posted screenshots of the updated client theme on FOSSBilling's Discord.

Hints From Belle

Belle said to me, about FOSSBilling, "If someone is going to use it, they should be technical and ideally be able to handle some PHP development."

Speaking of updates and changes, Belle advises FOSSBilling users to review the FOSSBilling changelog before updating. Changes occasionally affect people with customized installations, but people "using FOSSBilling as-is out of the box shouldn't really have such issues."

Last, but not least, Belle also advises reading the FOSSBilling FAQ.

Conclusion

FOSSBilling quickly and efficiently handled sign up, ordering, invoicing, and passing me to HestiaCP for provisioning. Support tickets and transactional emails from HostByBelle all worked as expected. FOSSBilling successfully kept track of payment and support transactions and also showed the history of each in clear interfaces. With HestiaCP, I quickly installed my above linked html test pages at HostByBelle and at YetiNode. Security certificates, like everything else, were easy.

Belle was super friendly and helpful via the HostByBelle FOSSBilling Support Tickets. @AuroraZero was, as always, wonderful, both via LES PMs and via email. Both HostByBelle and YetiNode are recommended for shared hosting.

FOSSBilling worked excellently from the perspective of a tester of shared hosting. The new era of a working, free, and open source billing platform has begun!

"Exited and prout to be posing my fist ever particle on LES."

@AlwaysSkint

Table Of Contents (T.O.C.):

1. Introduction and a bit of history
2. Hard Disk working principle
   2.1. CMR vs SMR recording technology
3. The physical size
4. Connection interface standards
   4.1. ATA
   4.2. SCSI and SAS
   4.3. SATA
   4.4. NVMe
   4.5. External drive connection standards
5. Hard disk performance basics
6. Hard disk technical specs
   6.1. Cache memory size
   6.2. Platter rotation speed
   6.3. Seek time and latency
   6.4. Sequential and random data transfer speed
   6.5. Durability stats
   6.6. Physical characteristics
   6.7. Capacity
   6.8. Air vs Helium
7. Hard disk buying recommendations
8. Conclusion

1. Introduction and a bit of history

The 10-second attention span people who like to "just skim" articles might want to skip straight to the chapter 2.

Relja BrevityIsNotMyVirtue Novović

There was a time when people used tapes to store large amounts of data. The main downside of using a tape for storage is the difficulty of finding a certain file, without reading the entire tape - (re)winding only gets you so far. It was much like playing a music cassette tape or a gramophone record.

In 1957, IBM built the first commercially available hard disks. HDDs allowed people to read, edit, delete, or add data to any file stored on the drive "right away." This is called "random access" and we take it for granted nowadays, but it was a major breakthrough in storage technology.

Over the following two decades, tape storage fought hard, but eventually lost in the vast majority of categories. There still are use cases where tape storage is the optimal choice, but that is a topic for a separate article.

Of course, the first hard disks were behemoths, used in enterprise server rooms and resembling modern day washing machines more than anything else. Their 24 inch large media was spun using powerful motors. Fun fact: my father and I put one such hard disk motor on our lawn mower, back in the 90s when there was a fuel shortage in my country. That thing is still running nicely at the time of writing, using three-phase 50 Hz 220V power.

An old hard disk motor - 1 horsepower, 3-phase power supply.
...For the life of me, I can’t remember if it was from a Sperry-Univac, or a decade younger Honeywell-Bull system...
Picture 1

Compare that to today’s 3.5" hard drives (and their smaller 2.5" laptop-friendly cousins). We’ve certainly come a long way. Nowadays, Solid State Drives (SSD) have replaced the traditional hard drive as the norm for most desktop computers and laptops. However, for storing large amounts of data, the good old hard disks are pretty much alive and kicking - with some more tech. improvements over the past few years. They are the stars of this article.

- T.O.C. -

2. Hard Disk working principle

I’ve seen programmers who don’t understand these concepts make an application that saves thousands of several-byte-small log files (i.e. very small files), and then get surprised to see the application take a huge amount of storage space. So, despite countless super-fast online programming courses, it helps to put some effort into understanding the hardware your stuff will be running on.

Relja Novović - shower thoughts

Let me briefly touch upon how these things work. A minute spent on the following text and pics should help you more easily understand hard disk technical specs. Ready?

Hard disks, as their name says, have discs covered in magnetic material which can be magnetised to north (N) or south (S) polarity. These discs are also called platters. There can be two or more platters in a hard disk drive (modern 3.5" drives can have up to five platters).

Similarly to a gramophone, the platters spin, and there are heads that can move across their surface to perform reading or writing.
Thanks to the damn hipsters, I hope that even the younger readers know what a gramophone is…

Hard disk drive's basic parts drawing
Picture 2

In the picture above, the DISK spins at over 5,000 rpm, while the R/W HEAD moves left-right, floating just about 10nm above its surface (and reading or writing data to it by using electromagnetic induction). When a hard disk drive is turned off, its heads are locked in place so they don’t hit the platters if the drive is transported. This is called parking the heads.

This part serves to introduce you to the technical terms you may hear: cylinders, tracks, and sectors. Alas, when you format a hard disk drive, you are logically dividing each platter into concentric tracks (imagine race tracks at the Olympics), and dividing each track into sectors (smaller sections along the track).

Every sector contains a set number of bytes (groups of eight digits that can only be a logical one or a logical zero). It’s usually 512 bytes on most file systems. Several sectors are grouped into a cluster, the smallest amount of disk’s storage space that can be taken by one file. If your cluster is, say 4096 bytes size, and you save a file that is only 1024 bytes large, it will still take up the entire cluster. However, if a file is 8192 bytes large, it will span across two 4094-byte clusters.

Before I totally confuse you by explaining what a cylinder is, a picture will explain it better:

Hard disk's logical division: cylinders, tracks and sectors
Image source: Wikipedia
Picture 3

So, now you clearly understand what a cylinder is: a set of tracks on the same distance from the platter’s centre, across all the platters in the drive. Cylinders are no longer used for addressing hard disk storage - they have been replaced with logical block addressing (LBA), but that's a topic for a separate article.

Here's what that looks like in real life:

Hard disk internals photo
Picture 4

Well, I hope you are now smarter than you were this morning. Let us move on to the more “flashy” stuff.

- T.O.C. -

2.1. CMR vs SMR recording technology

This one is rather important, especially if you value performance and reliability.

Remember what we’ve just said about tracks and how those are created in concentric circles on the platters when you format a hard drive? Each track contains sectors where bits are written and read from. That’s the old style, called CMR (Conventional Magnetic Recording).

Fact: you need more room (more track width) to write a bit to a track (i.e. polarize its magnetisation) than you need to read from it. That is why some genius thought it was a jolly good idea to have the tracks overlap! That is how SMR (Shingled Magnetic Recording) drives were born.

CMR vs SMR system of writing and reading data from a hard disk's platter
Picture 5

SMR lets you write about 25% more data on a given platter size, which lets you build larger capacity drives at a lower cost. What could possibly go wrong?

Well, SMR relies on the fact that the head for reading can work with a narrower track. But when you need to write to a track, you will also rewrite the adjacent tracks. So, if you write data on an SMR drive track, and its adjacent tracks have some data, you would need to first read and rewrite that data too, in order to prevent data corruption. This makes SMR drives perform awfully poorly when writing if they are loaded with data.

Some manufacturers don't disclose when their drives are built using the SMR system. Make sure to double check and avoid buying such drives - at least that's what I do and recommend.

- T.O.C. -

3. The physical size

3.5 inches is quite large.

Japanese proverb

Today, hard disks come in two formats: 2.5" and 3.5".

2.5" are the smaller ones with platters that spin slower, make less noise, heat and use less power. They are designed for use in laptops and portable (external) hard drive cases, and a part of this design is a more rugged mechanism of parking drive heads to avoid any damage in case of bumps or similar. Thanks to their lower power consumption, they can use a USB cable for power supply (handy for portable drives). An exception are server-grade 2.5" drives built to be compact, but work 24/7 in servers. Either option is more expensive per TB of storage space compared to 3.5" drives.

3.5" is the usual size for drives designed to be placed in desktop computers and many servers and NAS drives. These are generally faster, louder, require more power, but offer higher storage capacities for the price.

- T.O.C. -

4. Connection interface standards

The nicest thing about standards is that there are so many of them to choose from.

Ken Olsen

Gotta plug it in somehow, right? Let us go over some of the most widely used standards, starting with a couple of obsolete ones. Note that the given data transfer speed info is related to the connection's capacity, it doesn't mean every (or any compatible) drive can reach that speed.

- T.O.C. -

4.1. ATA

An obsolete standard. ATA (Advanced Technology Attachment) is the standardized name for what was also called PATA (Parallel ATA) and IDE (Integrated Drive Electronics). It uses a parallel connection, meaning several bits are transferred all at once. That is why ATA cables are flat and very wide.

ATA data transfer speeds go up to 133 MB/s.

- T.O.C. -

4.2. SCSI and SAS

SCSI (pronounced as “scuzzy”) stands for “Small Computer System Interface.” The standard started gaining popularity during the ‘80s (20th century) as a fast, high-quality standard for enterprise storage, because it allowed a connection of a relatively large number of drives, often placed in RAID (Redundant Array of Inexpensive Disks).

It started as a high-speed parallel connection, but got updated over the years. Its latest version uses a serial connection. It’s called “Modern Serial Attached SCSI” (SAS) and the latest models reach data transfer speeds of up to 1,875 MB/s. This stuff is (and always has been) expensive, and can be found in servers.

SAS interface can also accept SATA drives (though SATA interface won’t accept any SAS drives).

- T.O.C. -

4.3. SATA

SATA (Serial Advanced Technology Attachment). This is the de facto current desktop and laptop storage connection standard. It uses a serial connection at a high frequency (and speed), and allows for thin and long (up to one metre) cables.

SATA data transfer speeds go up to 600 MB/s (the latest, SATA III standard).

This interface is fast enough for most consumer-grade hard drives. If you are buying a HDD today, and reading this, you are most likely looking for a SATA HDD.

- T.O.C. -

4.4. NVMe

NVMe (Non-volatile Memory Express) was originally designed to work with SSDs (Solid State Drives). Unlike other storage media interfaces, NVMe lets you connect your drives directly to the CPU, using the PCIe interface. This allows for very low latencies and huge data transfer speeds of over 10,000 MB/s over PCIe 5.0.

At the time of writing, it is mostly used for SSDs (using M.2 connection format), though some companies, like Seagate, are working on building a HDD that natively connects via NVMe.

If you’d like to know more, Western Digital’s blog has a great article on NVMe.

- T.O.C. -

4.5. External drive connection standards

Let’s do a quick overview of the connection standards for external drives (portable drives, NAS expansion units, and similar).

FireWire
A standard launched by Apple, which has now become obsolete for reasons beyond the scope of this article. FireWire 800 standard could reach data transfer speeds of up to 100 MB/s.

USB
Current standard for connecting many external 2.5” drives is USB 3.1 with data transfer speeds of up to 1,250 MB/s, while the newer USB 3.2 can go up to 2,500 MB/s (older USB 1.0 and 2.0 standards are now practically obsolete).

Even newer USB4 standard comes in 20 Gbps (that is giga-bits, not giga-bytes per second) and 40 Gbps (5,000 MB/s - that is mega-bytes per second) variants (the 40 Gbps version requires the more expensive 40 Gbps cables).

There should also be a USB4 v 2.0 standard with speeds of 80 Gbps (10,000 MB/s), but I have never seen any appliances or even cables for that one.

Note, there is a USB C connector (physical connector) standard that often gets confused with the transfer speed standards of USB 3.1, 3.2, USB4 etc, but you technically could have the older, slower USB 3.1 working over a USB C physical connector.

USB is capable of supplying power over the same cable. USB A connector cables can supply up to 7.5W, while USB C connector cables can supply up to 100W or even 240W with the latest USB-C PD connector standard.

Thunderbolt
This standard requires that your motherboard supports it (which is not as commonly the case as it is with USB 3.2 nowadays). It offers speeds of up to 40 Gbit/s (5,000 MB/s).

It connects by combining the PCIe and DP (DisplayPort) signals into two serial signals, and also provides the needed power supply of up to 100W - all using the same cable.

In practice, Thunderbolt connectors and equipment are not very widely used, at least in my experience.

eSATA
Another standard that is getting obsolete (though Synology uses it for connecting some of their NAS expansion units). It doesn’t supply any power (so you need a separate power supply for the eSATA drives), though there is the eSATAp standard which does.

In its latest & greatest version, eSATA standard provides data transfer speeds of up to 600 MB/s.

I think this covers the standards you are most likely to encounter. Let us move on.

Connector types overview
Picture 6

- T.O.C. -

5. Hard disk performance basics

A child of five would understand this. Send someone to fetch a child of five.

Groucho Marx

Hard disks use spinning platters to store data. A read/write head moves up-down over the platter surface (see picture 2 for details). That is why they can read or write a lot faster if they are doing it all in one go. That is called sequential access reading/writing. If, on the other hand, they need to read a file (or files) dispersed over several non-adjacent sectors, that kind of reading/writing, called random access, is much slower.

Likewise, because reading only requires to, well, read the magnetic polarity of bits, while writing requires actual magnetisation of each bit, hard disk read speeds are a lot faster than their write speeds.

So, even if you buy the latest, greatest and fastest hard disk drive in the world, you should expect its random access speeds to be lower than its sequential access speeds, and its writing speeds to be lower than its reading speeds.

Manufacturers often state the most “optimistic” scenarios in their selling brochures, so it’s good to keep this in mind and double-check.

- T.O.C. -

6. Hard disk technical specs

There is no amount of memory that human stupidity can’t fill up.

Živojin Novović, when we discussed our home PC’s hard disk upgrade

In this “chapter,” I will cover the basic hard disk drive technical specifications, explain what each one means, how it affects the overall performance, and what its ballpark values are. I’ll use SATA drives as examples, since those are the most popular consumer and “prosumer” drives at the time of writing.

For brevity, I will not discuss every existing aspect of hard drives, but concentrate on the stuff you can quickly check and compare when deciding which drive to purchase.

6.1. Cache memory size

I have already explained what caching is (the context of that article was about website page caching, but the principle is the same). It boils down to using fast storage, such as RAM (Random Access Memory), to quickly store and retrieve data.

For a hard disk, this means: when you want to save a file, you just send it to the drive’s cache and call it a day, letting the drive worry about writing it to its platters for a permanent storage.

A similar thing happens when you are reading data from the drive. Say you open the document “pigeon-deliveries-june.doc”. Drives have smart controllers that can predict the likelihood of you requesting a file such as "pigeon-deliveries-july.doc" next, letting them take the time to read that file and load it into the drives cache, so you can retrieve it quickly if you decide to.

Caching, especially for reading files, is a hit-and-miss, but it can often help provide a faster perceived performance - i.e. despite the drive having the same read speed, you will effectively read and write your files faster, thanks to the use of cache.

Of course, cache memory is a lot more expensive than platter storage space, so even 1,048,576MB (about 1TB) large drives often come with only 64 MB of cache. That is why cache can’t help when reading or writing large amounts of data. Having said that, since hard disks are the slowest when writing, and most users write (save) a lot of smaller files at irregular intervals, cache helps a lot in practice.

Cache size usually depends on the drive's total storage capacity (larger drives have more cache memory). Typical cache size ranges from 32MB for lower end 1TB drives, to 256MB for larger capacity or high-end drives. More cache costs more, but manufacturers usually provide the optimal ratio based on drive’s capacity and intended use.

- T.O.C. -

6.2. Platter rotation speed

The faster drive’s platters spin, the faster you can read/write data (sequential read/write), and the less time it takes for the sector with the needed data to reach the reading/writing head (for random access). However, higher rotation speed also requires more power, and creates more heat and noise.

Typical speeds are 5,400 and 7,200 RPM, though some enterprise (SAS) drives spin up to 15,000 RPM.

Because of the extra noise, faster needn’t always be better, it depends on your use case.

- T.O.C. -

6.3. Seek time and latency

We know that read/write heads move across the platters. This takes time. Moving from one track to the next is a lot faster than moving from the outermost all the way to the innermost track. So, we have:

• Track-to-track seek time: often below 2ms.
• Average seek time: often about 5ms.

For some reason, seek time specs are next to impossible to get for most modern drives.

Now, since the platters are spinning, it might take some time for the sector you need to come under the reading head. Average time taken for this to happen is called the average rotational latency. For a drive that spins at 5,400 RPM, that is 60 (seconds) divided by 5,400, divided by 2 (to get the average). So, about 5.55ms. This goes down to 4.16ms for 7,200 RPM drives, and as low as 2ms for the 15,000 RPM drives.

Do not mix this rotational latency with the average read/write latency, which is:

• how long it takes for the drive's controller to figure out where the required block of data is on the platters,
• move the read/write head to it,
• and start data read/write process (transfer).

Average read latencies often go to about 30ms (and max. latencies can be as high as up to one second), while average write latencies can be ten times longer, so about 300ms.

These stats affect random read/write performance, but don’t affect sequential read/write performance.

- T.O.C. -

6.4. Sequential and random data transfer speed

Data transfer speed provided by manufacturers is basically average sequential read/write speed. It is often around 200 MB/s for modern drives. Drives that spin faster usually have a bit higher "data transfer speed" (i.e. average sequential read/write data transfer rate).

Random read/write performance gives data transfer speeds that are about 1% (one percent) of the sequential read/write speeds (so about 2 MB/s). These are often expressed in IOPS (Inputs and Outputs Per Second). In another article I've explained IOPS in more detail.

IOPS results differ based on the size of data blocks being read or written, but let's say that ballpark values are in hundreds of IOPS, and that with modern drives, write performance gives more IOPS thanks to the good use of caching. When a disk reads, it needs to read the given files wherever they are, often doing a lot of back-and-forth. When it writes, the data is stored in cache and then written in a more optimized way (less "random" write head movement).

- T.O.C. -

6.5. Durability stats

These are the stats related to durability. Note that these stats have more to do with the drive’s intended use than its actual quality (I’ll go to recommend models later on).

Power on hours per year
8,760 is for drives designed to work 24/7 (NAS and enterprise/server drives).
2,400 is for drives intended for personal or business computers (about 8 hours of daily use).

Unrecoverable read errors per bits read (URE)
This is basically the probability of a drive not being able to read a piece of written data (even with error correction attempted). 1 bit errror in 10^14 bits read is the usual number for consumer grade drives, while 1 in 10^15 or 10^16 is more common for NAS and enterprise/server grade drives.

In theory, 1 in 10^14 means there will be an error for every ~ 12 TB of data written, but as I said at the start of this “chapter,” this info has more to do with a drive’s intended use and manufacturer’s warranty, than it does with the actual durability and reliability of the drive.

Workload Rate Limit (WRL)
How many terabytes (TB) per year you can read or write to the drive. Consumer grade stuff is good for about 50TB, while the enterprise stuff is rated at 150TB per year or more.

Mean Time Between Failures (MTBF)
The arithmetic mean (average) of time between failures. Usually expressed in hours, and ranges around one million. Based on my knowledge and experience, this metric can be disregarded.

A stat that is actually useful (but it is not provided by drive manufacturers) is the Average Failure Rate (AFR). Unfortunately, to get this data, you would need a ton of input from service centres around the world. Or rely on Backblaze quarterly stats (Backblaze 2023 Q2 stats link) if they have used a drive that you are looking to buy.

- T.O.C. -

6.6. Physical characteristics

Apart from the size (form factor) explained in chapter 3, these are some other important disk drive physical characteristics.

Power consumption
How many watts the drive consumes when working on average (from about 5W to 10W or more for the larger enterprise grade drives). Faster spinning drives consume more power. Air filled drives (as opposed to helium filled ones) consume more power, since air creates more drag.

Noise (Acoustics)
Drives make noise when they are spinning, and even more noise when they are reading or writing. This noise level is expressed in A-weighted decibels (dBA). dB scale is logarithmic, meaning a 10 dB increase means 10 times louder, while a 20 dB increase means 100 times louder (10 times 10).

For example, 20 dBA is whisper, 60 dBA is normal conversation, and 100 dBA is a loud motorcycle.

Most drives are below 30 dBA when idling, and above 30 dBA when reading. But even a couple of dBA difference is noticeable, as explained in the previous paragraph. 20 dBA idle and 25 dBA when reading/writing is considered to be very quiet for hard drives.

- T.O.C. -

6.7. Capacity

Modern hard disk storage capacity is expressed in terabytes (TB). TB is a thousand megabytes (MB). Some smaller models have sizes of 500 MB or similar, but those are getting obsolete. Modern drives range from one to a dozen or more TB.

Larger capacity drives often have more platters, consume more power and cost more.

- T.O.C. -

6.8. Air vs Helium

For decades, hard disks had a small hole (with a dust filter) to let the air move in and out (as the inside air temperature changes). Read/write heads float at about a dozen nanometres above the platters. If the air becomes too thin (at high altitudes), air-filled drives will not work properly, their heads will drop and scratch the platters.

Helium, unlike air, is more stable and provides a lot less drag as the drive platters spin (and its read/write head moves).

The problem with helium is that its molecules are so tiny, that it is very difficult to keep it from seeping out of the hard drive’s case. Some extraordinary high tech. advancements over the past decade have allowed manufacturers to seal their hard disk drive units well enough for the damn gas to stay inside. And voila, today we have helium filled drives.

Helium filled drives make less noise, use less power (all thanks to the lower drag), and, because they are completely sealed, can operate safely in low pressure environments. Making the tight seal costs more, so at the time of writing, only high-capacity drives (10TB or more) come helium filled.

- T.O.C. -

7. Hard disk buying recommendations

I am not rich enough to buy cheap stuff.

English proverb ( if this one doesn't get me banned, nothing will! )

In this article I'll say what I would buy (or have bought). I am not in a position to test dozens of models, but I work with computers for a living, and I like listening to people: service technicians, system administrators (and the good folks of the LowEndSpirit forum ).

How big?
If at all possible, for sizes of 2TB or smaller, do not buy a hard disk drive, get an NVMe or at least a SATA SSD. Today, hard disks make sense for 4TB or larger sizes.

For a NVMe SSD, I use and recommend Samsung EVO 970 Plus (Amazon affiliate link). It costs well under $100 for the 2TB version, it is super-reliable and far from slow, even under high loads. Note that it supports PCIe 3 speeds only, but for most use cases that means less heat, and performance that is more than good enough.

PCIe 4 gives a sequential read/write speed boost due to a higher bandwidth, but for general use (i.e. many random read/write operations) it gives next to no benefits (also, PCIe 3 NVMe drives are also pretty fast even for sequential read/write).

My opinion:
At the time of writing I can not recommend a PCIe 4 (or PCIe 5) NVMe drives in good faith. Many models of many manufacturers have problems with heating and reliability - and as the heat rises, you throttle back to PCIe 3 speeds or lower. Samsung has a price premium which is more than worth it for the above-recommended EVO 970 Plus model, but their PCIe4 drives had problems and I'm still not sure how reliable they are despite the firmware updates published after a public outrage.
UPDATE, December 2023:
Samsung 990 PRO Heatsink (Amazon affiliate link) looks promising (I’m yet to test it personally). Hardware-supported encryption, and, as far as I know, no firmware-related problems that plagued some Samsung NVMe models. It comes in a 4TB version (along with the 1TB and 2TB alternatives), and a built-in heatsink (good cooling is important for PCIe Gen4 speeds). If I were buying a new system drive today, I’d go with the 4TB version of this one.

Which brand (manufacturer)?
I'll start with Toshiba. Their drives are on the noisier side, but I've had great results with them in terms of reliability and durability. Batches that are sold in my country (and Hungary for that matter) seem to have the lowest percentage of "lemons."

Seagate is another decent manufacturer I've had good results with.

Western Digital (WD) is a highly renowned manufacturer, but I have not had very good results with them.

The thing is: I am only one person. Sure, decades of experience, but that still doesn't make for a statistically useful data. Also, many manufacturers make a blunder every now and then. So I would argue that the manufacturer is not nearly as important as the particular model. At the time of buying, see which models perform well and are reliable. If you can normally go and talk to people who deal with warranties of large hardware stores, they will probably be able to tell you which models get the lowest number of returns (though, keep in mind that expensive models get sold a lot less, so them having fewer returns doesn't always mean they are more reliable).

Concrete model recommendations
See the paragraph above to understand the limitations of recommendations like this. With that out of the way, you should also know that most manufacturers have turned to SMR recording technology for their consumer-grade drives. I would recommend you avoid SMR drives. They are not worth the savings.

My opinion:
That makes perfect sense in capitalism: making the low and mid-priced stuff be less durable and less reliable, and practically "forcing" people to buy the more expensive stuff.

For larger storage capacities, I would recommend Seagate IronWolf 12TB NAS (Amazon affiliate link - 'cause yachts won't pay for themselves! - though for the US folks, I would recommend buying directly from the manufacturer to get a better and longer warranty protection) or higher capacity IronWolf models if you need more storage.

If you wish to save some money or don't need that much storage space, you could go with a smaller, Seagate IronWolf 4TB NAS drive (Amazon affiliate link).

Alternatively, if you don't mind a bit more noise, you can't go wrong with Toshiba N-series drives, like Toshiba N300 4TB NAS (Amazon affiliate link).

- T.O.C. -

8. Conclusion

As with many of my other articles, this one too serves primarily as my own reminder and reference. I will try to keep it up-to-date and correct any errors - my articles aren't written by AI or "content writers," and English is not my native as you have probably figured out by now.

If you have questions, additions and, especially, corrections - please post a comment.

- T.O.C. -

Relja Novović - about the author:
https://www.bikegremlin.com/about/

What?

Plan 9 from Bell Labs is an operating system from the creators of the C Programming Language and the Unix operating system.

Although Plan 9 revolutionized many aspects of Unix, it nevertheless still remains possible in 2025 for Unix fans to access aspects of old Unix by installing Plan 9 inside Qemu and changing the Plan 9 boot procedure so that Plan 9 stays in the terminal rather than starting a graphical monitor.

Why?

• This method provides a text-only terminal with "infinite" scrollback. Infinite scrollback is how paper terminals used to work before the curses interface and the Graphical User Interface ("GUI") were invented.

• Plan 9 runs on the i386 processor architecture in addition to many other architectures.

• Plan 9 has very few system calls compared to other operating systems.

• Plan 9 comes with full source code and build tools.

• The code for Unix userland utilities which remain in Plan 9 is shorter than later versions.

• Plan 9 is fast!

Why not?

• This method, which perhaps involves the conflation of text and terminal, is arguably a complete misunderstanding of everything that's truly the greatest about Plan 9. See this excellent discussion (but be a little careful when reading, because possibly not everything that's said is 100% correct).

How?

Download Plan 9. Options include the classic Plan 9 from Bell Labs (used here, see links just below), as well as, probably, 9legacy and 9front, neither of which were tested for this post.

wget http://9p.io/plan9checksums.txt 
wget http://9p.io/plan9/download/plan9.iso.bz2

# md5 sum for the bz2 file used here:
# 5183b2d0f11fded871461416f8c78cfb  plan9.iso.bz2

Make a qemu disk image for our Plan 9 install.

# qemu-img create -f qcow2 plan9-console.qcow2 10G

Use your favorite VNC app set to localhost:5900. Relaunch the app if it crashes when Plan 9 switches from terminal to grraphics.

# ssh tom@toto -L 5900:localhost:5900 # VNC via ssh tunnel to your server from your local machine.

Install Plan 9. Usually it's easy, just boot the ISO and accept the defaults. There are lots of tutorials and videos about the installing Plan 9. Here below is a qemu command to start the install. This command doesn't use KVM. Not using KVM makes slower output, which is easier to read, but you might need to wait a minute for longer operations. If you wish, add KVM with -enable-kvm.

qemu-system-i386 \ 
  -vnc 127.0.0.1:0 \ 
  -hda plan9-console.qcow2 \
  -cdrom plan9.iso \
  -boot d

After Plan 9 is installed, you see the graphical Plan 9 (screenshot above). Edit plan9.ini with the Plan 9 VT-100 emulator and the ed editor.

# In the Plan 9 rio terminal, type vt
# In the vt terminal, type '9fat:' (Don't forget the colon!)
# ed /n/9fat/plan9.ini : 
#   - remove last two lines (monitor, and size)
#   - add a line which says console=0

After you change plan9.ini, you don't need VNC, just ssh into your server and run Qemu.

qemu-system-i386 \ # Not using KVM makes slower output, which is easier to read.
  -nographic \
  -hda plan9-console.qcow2 \
  -boot c

Here below is the output of the above. We can launch the Plan 9 versions of old Unix commands like awk and ed. We can get infinite scrollback, almost like a paper terminal.

tom@toto:~/plan9$ cat 20250228a-boot-messages
# 20250228 Boot Plan9 in Qemu with plan9.ini changed to remain in host node shell

tom@toto:~/plan9$ ./start-plan9-console

SeaBIOS (version 1.16.3-debian-1.16.3-2)


iPXE (https://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+06FC6F60+06F06F60 CA00



Booting from Hard Disk...
MBR...PBS1...
Plan 9 from Bell Labpcirouting: BIOS workaround: PCI.0.1.3 at pin 1 link 96 irq 10 -> 9
 disk loader

cpu0:  3091MHz GenuineIntel Celeron (cpuid: AX 0x0663 DX 0x781ABFD)
ELCR: 0C00
497M memory: 497M kernel data, 0M user, 18M swap
found partition #S/sdC0/data 0 20,971,520
disks: sdC0 sdD0
trying sdC0....found 9pcf
.1121326.....................................................................................................0
entry: 0xf0100020

Plan 9
E820: 00000000 0009fc00 memory
E820: 0009fc00 000a0000 reserved
E820: 000f0000 00100000 reserved
E820: 00100000 07fe0000 memory
E820: 07fe0000 08000000 reserved
E820: fffc0000 100000000 reserved
cpu0:  3096MHz GenuineIntel Celeron (cpuid: AX 0x0663 DX 0x781ABFD)
ELCR: 0C00
igbe: p->cls 0x0, setting to 0x10
#l0: i82543: 1Gbps port 0xFEB00000 irq 11: 525400123456
128M memory: 54M kernel data, 74M user, 297M swap
usbinit...usbd.../boot/usbd: /dev/usb: no hubs
no /srv/usb...no usb disk...pickmethod...read #e/nobootprompt...read #e/bootargs...outin...root is from (tcp,[ . . . copy paste error] 
pickmethod done
bind #æ...bind #S...partinit...auth...user[none]: glenda
usbinit...usbd.../boot/usbd: /dev/usb: no hubs
no /srv/usb...no usb disk...mount usbd...boot: can't open /srv/usb: '/srv/usb' file does not exist
time...
fossil(#S/sdC0/fossil)...version...can't stat /srv/partfs.sdXX: '/srv/partfs.sdXX' file does not exist

init: starting /bin/rc
rio: can't open display: initdisplay: /dev/draw/new: no frame buffer
init: rc exit status: rio 30: display open

init: starting /bin/rc
term% who am i
glenda
term% awk 
Usage: awk [-F fieldsep] [-mf n] [-mr n] [-v var=value] [-f programfile | 'program'] [file ...]
term% ed
a
hello
.
w hello
6
Q
term% ls -l hello
--rw-rw-r-- M 8 glenda glenda 6 Feb 28 21:07 hello
term% fshalt
syncing.../srv/fscons...prompt: 
halting.../srv/fscons...
prompt: 
done halting
Ctrl-a x # Quit qemu
tom@toto:~/plan9$ 

Linveo kindly provided several free VPSes to guys actively posting in the LES BSD Thread. Linveo also kindly added @cmeerw's NetBSD 10 Minimum image. As shown above, NetBSD 10 Minimum now easily can be installed at Linveo directly from "Other BSD" category on the Rebuild page of vmcontrol.linveo.com.

After installing @cmeerw's NetBSD 10 Minimum image, with help from the NetBSD Guide references linked below, I followed the steps listed below to upgrade from NetBSD 10 to self-compiled NetBSD-current.

Getting to self-compiled NetBSD-current requires us to add to the NetBSD 10 Minimum VPS a NetBSD 10 compiler, other build tools, and the source code for both the NetBSD-current kernel and the NetBSD-current userland. Then a NetBSD-current compiler and other build tools are compiled by the NetBSD 10 compiler. Next, the newly compiled NetBSD-current build tools are used to build the NetBSD-current kernel and the NetBSD-current userland. The end result is a NetBSD-current VPS running its own code as compiled on itself by its own compiler.

My Linveo VPS seems to reboot and to run normally following the upgrade to NetBSD-current. There seems to be plenty of space remaining available, though the X sources and pkgsrc haven't yet been added:

linveo# df -h .
Filesystem     Size   Used  Avail %Cap Mounted on
/dev/dk2        48G    10G    35G  22% /
linveo# 

linveo$ CVS_RSH=ssh cvs -danoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P pkgsrc
  [ . . . ]
linveo$ df -h .
Filesystem     Size   Used  Avail %Cap Mounted on
/dev/dk2        48G    11G    35G  24% /
linveo$ 

• Issue

The biggest issue that I know about with respect to the NetBSD-current upgrade proposed here relates to the etcupdate command, which is the last command in the sequence below.

etcupdate is an sh shell script. There also is a man page, which mentions that the idea for etcupdate came from FreeBSD's mergemster.

I need to figure out how to set options making etcupdate faster and easier to run within the limited scope of this single CPU architecture NetBSD-current upgrade. The VPS is not attempting to build the full multi-architecture NetBSD panopoly. Perhaps limiting the etcupdate source directory to my VM's amd64 architecture might help. Perhaps -- as recommended in the man page -- running etcupdate on a tar file instead of directly on the sources also might help.

• NetBSD 10 Image

Here are links to @cmeerw's NetBSD 10 image and to a discussion of how the image was made:

https://lowendspirit.com/discussion/comment/187386/#Comment_187386
https://lowendspirit.com/discussion/comment/187452/#Comment_187452

• NetBSD Guide Reference Links

Here are reference links to the very helpful NetBSD Guide:

https://www.netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-cvs-netbsd-release
https://www.netbsd.org/docs/guide/en/chap-updating.html

• Adding Compilers To The NetBSD 10 Image

Here is the LES comment where @cmeerw showed us the one line command used below to add NetBSD 10 compilers to the NetBSD 10 base image:

https://lowendspirit.com/discussion/comment/186774/#Comment_186774

• Recipe

Here are the steps I followed to convert a clean, new Linveo VPS install of @cmeerw's NetBSD 10 image to what seems to be self-compiled NetBSD-current:

• Run as root to install a compiler, create needed directories, and an unprivileged user

ftp -o - https://cdn.netbsd.org/pub/NetBSD/NetBSD-10.0/amd64/binary/sets/comp.tar.xz | progress -z tar xpf - -C /
cd /usr
mkdir obj src tools
mkdir /home
useradd -m -G wheel tom
passwd tom
chown tom obj src tools
su tom 

• Run as the unprivileged user which owns the obj, src, and tools subdirectories

export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
export CVS_RSH="ssh"
cvs checkout -A -P src
cd src
less UPDATING
./build.sh -j 2 -O ../obj -T ../tools -U distribution
./build.sh -j 2 -O ../obj -T ../tools -U kernel=GENERIC

• Back to root to install and reboot to use our newly compiled NetBSD-current kernel

mv /netbsd /netbsd.old
mv /usr/obj/sys/arch/amd64/compile/GENERIC/netbsd /
shutdown -r now

• As root, install and reboot to run the newly compiled NetBSD-current userland

cd /usr/src
./build.sh -j 2 -O ../obj -T ../tools -U install=/
shutdown -r now

• As root, postinstall

/usr/sbin/postinstall -s /usr/src check
/usr/sbin/postinstall -s /usr/src fix

• As root, etcupdate

/usr/sbin/etcupdate -s /usr/src

Please point out any mistakes.

Any questions, please ask.

Thanks @linveo for the nice VPS! Thanks @cmeerw for the NetBSD 10 Minimum image! Thanks to the NetBSD project! And thanks to LES!

Why

The gear I've got for firewall is stupidly overkill (optane drive & 16GB RAM) for something that is 99% idle so I want to stick a couple other things on the same hardware via proxmox.

Assumed knowledge and requirements

Proxmox, linux cli, basic networking, (very basic) firewall knowledge. Nothing here is super hardcore technically as such...just unforgiving in sequence & details if unfamiliar hence notes. i.e. when it doesn't work it's not obvious where you fkd up.

Bring backups & loads of time. Frankly...assume that you'll fu.ck this up if you haven't done this before. So this is a Saturday morning project not a Sunday night one.

Before you start set up a ventoy USB stick adding both proxmox and opnsense ISO on there (plus perhaps a PDF copy of this post). You may not have internet access while doing this so come prepared...

I'm using my own IP ranges etc in post - adjust as needed. Key pieces you need to know to follow my descriptions:

vmbr0 is LAN side
vmbr1 is WAN (internet) side. (Proxmox sticks VMs on 0 by default so you want LAN side being 0)
10.32.0.1 is the opnsense VM [LAN side], 10.31.0.5 is WAN side but that doesn't really matter key part is 31 makes it separate from our main 32 LAN
10.32.0.2 is the LAN side IP of proxmox
My device has 5 ports with enp2s0 being internet facing [aka vmbr1], rest of the NICs are bridge to vmbr0
The key networking file on proxmox is at /etc/network/interfaces

Assumes upstream ISP is Eth based and just DHCPs whatever is connected. If you've got something else on ISP side you're on your own (sorry)

Virtualized FW

There are two basic ways to do this. Either pass through the NICs to the FW VM, or do two sets of bridges with one being WAN (internet) side and one LAN side. Passthrough is tricky to get working on a basic linux/bios/hardware level but solves some security issues. I'm doing the bridge approach because my gear (oddly) doesn't seem to support pass through. This guide is bridge only...so abandon ship now if you're going passthrough.

Bridge has one significant security implication. Proxmox gets the external traffic and bridges it to the firewall VM. Grand. Except Proxmox is fond of serving the proxmox management interface on ALL interfaces. So it's serving the damn management GUI before anything gets to our FW. Meaning it is potentially exposed BEFORE our fancy opnsense security gets to it. That's a problem we're trying to get around with below section on messing with the Proxmox firewall...despite setting up a opnsense firewall. [In reality the WAN side has a pvt non-routable IP so should be fine even without rules but I don't 100% trust that]

Keep this in mind when doing say incoming wireguard...you're effectively double-firewalled...and would need to pass through wireguard on both.

Security

The bulk of this guide happens BEFORE anything is connected to the internet, mostly because above "I don't trust this" issue. Just the FW device and a laptop directly connected to it.

GO

Step 1: Install proxmox & ensure you can access the GUI

On the network section do:
IP 10.32.0.2
Gateway 10.32.0.1
Set DNS to 1.1.1.1
Select a NIC that is LAN side (enp7s0 for me) - this should be whatever your laptop is plug in to

The proxmox interface is on https://10.32.0.2:8006

Step 2: Set up a second bridge in /etc/network/interfaces with WAN

auto vmbr1
iface vmbr1 inet static
        address 10.31.0.5/24
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

iface enp2s0 inet manual

Note how it is set to a static unroutable IP despite us aiming for DHCP upstream ISP side. That's because we want the firewall VM to grab the DHCP from ISP, not the proxmox level interface.

Restart to make the 2nd bridge show in GUI.

Step 3: Set up a opnsense VM

Copy over ISO off USB stick like so:

mkdir /media/usbstick
fdisk -l  [[[to get the /dev/ location for the stick]]]
mount /dev/sda1 /media/usbstick
cp /media/usbstick/OPNSense.sio /var/lib/vz/template/iso/

If the iso is in that location then proxmox GUI should pick it up.

Set up a VM that has 12GB, 5120mb RAM and 3 cores. Going lower on space & ram works but has consequences downstream on how zfs plays out so if you can do 12 & 5 or more.

Before starting VM add the second bridge (vmbr1). Ideally add the LAN side during VM creation so that the naming 0 & 1 line up consistently. Memorize the MACs on both (sorta...just which is which).

Step 4: Install opnsense

Where it asks you for a login, use installer as user and opnsense as password

Key point here is you need to line up what opnsense thinks is LAN side and WAN side...with the right MAC per above...which in turn maps to the right bridge...which in turn maps to a physical port.

Finish install, change root pass & unmount the ISO. Reboot.

Step 5: Initial config of opnsense

Log into console of opnsense (the proxmox built in one). Option 2 - set interface IP. Configure LAN side. This part is a little confusing. We DONT want it to use DHCP as in where it gets it's IP, but we want it to SERVE DHCP...so that other LAN devices can get an IP from it. So in this menu the first time it asks you about DHCP the answer is no, the second time the answer is yes. Set IP to 10.32.0.1 and 24 as subnet bit count. Enable DHCP server. Start clients at 10.32.0.51 and end at 10.32.0.254. That way we've got ~50 IPs for statics. Change GUI to http. Reset to defaults - yes.

...now try to access GUI on http://10.32.0.1

On the GUI you should have an initial setup. Set hostname, domain, and DNS. Go with a public one for now e.g. 1.1.1.1 Leave the WAN side as DHCP - no need to change anything here.

Step 6: Firewall

Go back to proxmox & navigate to datacenter on left then to firewall. This stuff cascades so the rules you set on datacenter level apply to the device and to VMs below.

The firewall rules use first match principle, starting with zero i.e. top of list. So in general we want a pattern like so

[0] - Allow SSH on LAN side
[1] - Disallow SSH on everything

So lan side incoming SSH hits rule 0 matches that and gets approved, while everything else fails rule 0 then goes to rule 1 and gets dropped.

Direction: In
Action: Accept
Interface: vmbr0 [[[[this is LAN side]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick

Direction: In
Action: Drop
Interface: [[[[leave empty]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick

Do same for 22 TCP and ICMP (protocol, no port). I'd suggest messing with ICMP to experiment and get a better grasp of what blocks when.

Triple check that you've got this right. Next step is enable firewall...and if you fkd this (8006 specifically) up then you're locked out and will need a physically screen & keyboard to sort it out.

You should also block these ports on vmbr1 (internet/wan) for good measure:
TCP 111, UDP 111, UDP 323, TCP 25, TCP 3128. No idea what they all do...but proxmox is listening on them & I don't want them open internet facing. I used command "lsof -i -P -n" to work out what ports are being listened to.

Step 7: Connect device to internet

If all went well then a device plugged into LAN side can now access the internet. First thing you want to do is head over to

https://www.yougetsignal.com/tools/open-ports/

And confirm they can't see your port 8006

Trouble shooting

1) You'll need to set a static IP in the right range on your laptop when connecting directly to a device. So when you can't reach the interfaces (proxmox or opnsense) checking that you're on the right net is helpful. Also windows is really shtty on this so sometimes you need to disable and reenable the network adapter to make it pick up things right. Try that before assuming something else is broken. You may also need to restart the laptop entirely...windows tends to get confused easily when rapidly switching networks.

2) Very likely that you'll f up DNS somewhere along the way...so when trying to work out whether a path through the firewall works try to use IPs like 1.1.1.1 with ping rather than google.com.

3) If genuinely stuck do ping device by device. i.e. my laptop is connected to a proxmox device...can i ping that by IP. Yes, ok can I ping opnsense by IP. Yes, can I ping 1.1.1.1. Yes. Can I ping google.com.

Devices

You should point all devices at the opnsense instance for both gateway and DNS [10.32.0.1 for me]. Never point a device at the pihole IP directly...that would work yes but it'll get you into trouble later in unexpected & hard to troubleshoot ways. You want the devices asking opnsense for DNS and opnsense in turn asking the pihole/adguardhome, and that asking 1.1.1.1 in turn. It's under Unbound in opnsense...section called Override...that's where the pihole goes. Also I suggest using Adguard home...it's superior to pihole.

YAY & Thoughts

Once it all sorta works, export the config in opnsense and save a copy of the proxmox /etc/network/interfaces - opnsense has a really nice "one config backup file everything is in it" config thing going...use it.

If anyone finds holes in this post please do tell...I'm running this live so if I'm wrong about something I'd like to find out from you rather than an evil hacker

Full /etc/network/interfaces for completeness

root@firewall:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp3s0 inet manual
iface enp4s0 inet manual
iface enp5s0 inet manual
iface enp6s0 inet manual
iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.32.0.2/24
        gateway 10.32.0.1
        bridge-ports enp7s0 enp6s0 enp5s0 enp4s0 enp3s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.31.0.5/24
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

And firewall view:

Introduction

Back on May 19, 2023, the 大 (big, great) @tang_cn wrote an awesome LES tutorial called How to Become an One-Man NAT VPS Provider?

@tang_cn's tutorial covers setting up NAT VPSes using the virt-install command line utility from libvirt.

Trying On A Debian 12.2 Node

I decided to try following @tang_cn's tutorial. The Node which @tang_cn's tutorial had used was running Debian 11. I tried with a Debian 12.2 Node. On Debian 12.2, I ran into two issues:

• @tang_cn's example code did not include a possibly newly required line --osinfo detect=on,name=OSNAME. Adding the osinfo detect=on,name=debian12 line resulted in a fatal error. So, yes, Debian is great, but Debian 12.2's virt-install did not include "debian12" among the list of supported releases shown by virt-install --osinfo list! I bypassed the error with --osinfo detect=on,require=off, but I am unclear on what, if any problems turning off osinfo require might have caused.

• After completing @tang_cn's install steps, my Debian guest VM worked fine over remote VNC via ssh tunnel into the VM's console. The VM could ping and could be pinged from the WAN over IPv6, but not over IPv4. I didn't solve the NAT IPv4 networking issue, whatever it was.

Trying On A Fedora Rawhide Node

I tried again on a Node runniing Fedora Rawhide. I did find that the more current but much less tested code in Rawhide enabled virt-install to support Debian 12 VMs.

• The Rawhide Node

The Rawhide Node is an Intel E-2276G rented from the Hetzner server auction for €33.70 monthly. This price is at least a couple of Euros above the minimum auction price for this server. Nevertheless, this server is worth keeping long term since it has two new NVMe disks. Not all auction servers have new disks.

• References

Fedora has an introductory Virtualization – Getting Started tutorial. The Fedora tutorial links to libvirt's wiki article on common libvirt networking configurations.

Let's start by following the command line versions of the steps in the Fedora tutorial.

• Step 1: System Requirements

Virtualization requires certain minimum RAM amounts and also CPU virtualization extensions, either Intel VT or AMD-V. The minimum RAM amounts are satisfied here since the Node has 64 GB DDR4 ECC RAM. The CPU passes the wiki article's egrep '^flags.*(vmx|svm)' /proc/cpuinfo check.

• Step 2: Install Virtualization Software

The Fedora dnf package manager's virtualization group on our Rawhide server still seems the same as that shown in the Fedora wiki.

[root@polonium ~]# dnf groupinfo virtualization
Last metadata expiration check: 3:24:19 ago on Sun 19 Nov 2023 03:19:10 PM UTC.
Group: Virtualization
 Description: These packages provide a graphical virtualization environment.
 Mandatory Packages:
   virt-install
 Default Packages:
   libvirt-daemon-config-network
   libvirt-daemon-kvm
   qemu-kvm
   virt-manager
   virt-viewer
 Optional Packages:
   guestfs-tools
   python3-libguestfs
   virt-top
[root@polonium ~]# 

Let's try installing with the optional packages.

dnf group install --with-optional virtualization

Whew! That was 481 packages, but the install took only about one minute. The entire 2,130 lines of terminal output was saved, so, if you want to see it, please ask.

Let's use systemctl to start and then to enable the libvirt daemon. Enabling the daemon makes it restart automatically when the Node reboots.

[root@polonium ~]# date -u; systemctl start libvirtd
Sun Nov 19 10:49:12 PM UTC 2023
[root@polonium ~]# date -u; systemctl enable libvirtd
Sun Nov 19 10:49:30 PM UTC 2023
Created symlink /etc/systemd/system/multi-user.target.wants/libvirtd.service → /usr/lib/systemd/system/libvirtd.service.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd.socket → /usr/lib/systemd/system/libvirtd.socket.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd-ro.socket → /usr/lib/systemd/system/libvirtd-ro.socket.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd-admin.socket → /usr/lib/systemd/system/libvirtd-admin.socket.
[root@polonium ~]# 

Verify that KVM is properly loaded.

[root@polonium ~]# lsmod | grep kvm
kvm_intel             425984  0
kvm                  1376256  1 kvm_intel
irqbypass              12288  1 kvm
[root@polonium ~]# 

• Step 3: Default Networking Support

The Fedora tutorial tells us to expect a 192.168.x.x subnet to be set up by default on the Node. The VM guests should be able to connect out to the WAN by using the Node's main IPv4 as a gateway. Connections from the WAN into the guests require firewall DNAT rules on the Node or a bridged environment.

Let's use the iproute2 suite to see what happened to the default network when we installed the dnf virtualization group.

[root@polonium ~]# ip link show
  [ . . . ]
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
[root@polonium ~]# 

It's apparent that installing the virtualization group added a bridge called virbr0. The bridge is down now, but will be put up when we install and start a VM.

• Step 4: Create A Guest With virt-install

Let's install a Debian guest VM. Even though we are following a Fedora tutorial. 🙈🙉🙊

First we get the guest's iso image, SHA512 sum, and signature. The Fedora tutorial put the iso images in /var/lib/libvirt/images/. I began by creating the images directory and then a separate iso directory inside /var/lib/libvirt/images. Libvirt seems flexible about the iso location. Because I use the iso images for additional purposes beyond libvirt, if I were doing this again, I might create the iso directory in /var/lib, independent of and on the same level as the libvirt directory.

Reference: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/

cd /var/lib/libvirt/images/iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.2.0-amd64-netinst.iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS.sign

Next, let's check the SHA512 sum and the signature.

Reference: https://www.debian.org/CD/verify

sha512sum -c SHA512SUMS --ignore-missing
gpg --keyserver keyring.debian.org --recv DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg --verify SHA512SUMS.sign SHA512SUMS

Now we get to use virt-install to create our Debian 12 guest. Here is the install command directly from the Fedora tutorial.

virt-install --name Fedora39 \
--description 'Fedora 39 Workstation' \
--ram 4096 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/Fedora-Workstation-39/Fedora-Workstation-39-20180518.0.x86_64.qcow2,size=20 \
--os-type linux \
--os-variant fedora39 \
--network bridge=virbr0 \
--graphics vnc,listen=127.0.0.1,port=5901 \
--cdrom /var/lib/libvirt/images/Fedora-Workstation-39/Fedora-Workstation-Live-x86-64-39-1.1.iso \
--noautoconsole

Here is the virt-install command which was used for this test. Note that --os-type has been deprecated, and that the version of virt-install, still numbered 4.1.0 in Rawhide, supports Debian 12.

[root@polonium ~]# cat -n /root/Install-Debian-KVM-Guest.sh
     1  virt-install --name Debian-12-Guest-1 \
     2  --description 'Debian 12 from netinst.iso' \
     3  --ram 4096 \
     4  --vcpus 2 \
     5  --disk path=/var/lib/libvirt/images/Debian-12-20231119.1.x86_64.qcow2,size=20 \
     6  --os-variant debian12 \
     7  --network bridge=virbr0 \
     8  --graphics vnc,listen=127.0.0.1,port=5901 \
     9  --cdrom /var/lib/libvirt/images/iso/debian-12.2.0-amd64-netinst.iso \
    10  --noautoconsole
[root@polonium ~]# /root/Install-Debian-KVM-Guest.sh

Starting install...
Allocating 'Debian-12-20231119.1.x86_64.qcow2'                    |    0 B  00:00:00 ... 
Creating domain...                                                |    0 B  00:00:00     

Domain is still running. Installation may be in progress.
You can reconnect to the console to complete the installation process.
[root@polonium ~]# 

virt-install leaves us with a guest which is running. The guest has a disk image qcow2 file and also is connected to the net-install iso. virt-install boots the guest from the iso. Thus, at this point, we have installed the guest on the Node, but we still need to install the guest OS inside the guest's disk image.

• Step 5: Install The Guest OS Inside The Guest

We connect to and install the guest in the usual way appropriate to the distribution that the guest is using. Inside the guest, the guest's IPv4 network is automatically configured by DHCP using the subnet IP assigned to the guest and using the IP of the Node as the gateway.

To connect to the guest from our local address, we set up an ssh tunnel from our local machine to the Node.

ssh root@$NODE_IP -L 5901:localhost:5901 

Then we connect via VNC. In our local VNC client, we enter the localhost IP address followed by the port number which was used in the virt-install --graphics vnc,listen=127.0.0.1,port=5901 \ flag. So, here, we would connect to 127.0.0.1:5901.

When we are finished installing the guest OS on the guest qcow2 image, we normally remove the iso and click reboot in the installer so that the guest reboots from the qcow2 image rather than from the netinst iso. However, here, with libvirt, the guest shuts down and does not reboot. Instead, we now need to start the guest with virsh.

• Step 6: Manage The Guest With virsh

virt-install leaves an XML file for each guest in /etc/libvirt/qemu.

To start a virtual machine, use virsh create with the full filename of the guest's XML file.

[root@polonium ~]# cd /etc/libvirt/qemu
[root@polonium qemu]# ls -l
total 16
drwx------. 2 root root 4096 Nov  6 00:00 autostart
-rw-------. 1 root root 7235 Nov 20 03:58 Debian-12-Guest-1.xml
drwx------. 3 root root 4096 Nov 19 22:49 networks
[root@polonium qemu]# virsh create Debian-12-Guest-1.xml
Domain 'Debian-12-Guest-1' created from Debian-12-Guest-1.xml

[root@polonium qemu]# virsh list
 Id   Name                State
-----------------------------------
 1    Debian-12-Guest-1   running

[root@polonium qemu]# 

To gracefully stop a guest:

# virsh shutdown <virtual machine (name | id | uuid)>

To destroy a guest which already has been stopped. The guest's qcow2 image needs to be removed manually.

# virsh undefine <virtual machine (name | id | uuid)>

Success!

Logging into the fully installed and rebooted guest via VNC showed IPv4 WAN connectivity from inside the guest.

Also, we can use ssh to log in to the guest from the Node.

[root@polonium ~]# ssh tom@192.168.xxx.xxx
The authenticity of host '192.168.xxx.xxx (192.168.xxx.xxx)' can't be established.
ED25519 key fingerprint is SHA256:0NfbDJkPk5Lt+dsFOlt/tWOq2+Edf/UPn136Yis9Kvo.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.xxx.xxx' (ED25519) to the list of known hosts.
tom@192.168.xxx.xxx's password: 
Linux debian 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Nov 19 22:44:07 2023
tom@debian:~$ 

Note: QEMU/KVM Without libvirt

We could have used qemu without libvirt. But that might not be so easy. Let's see the qemu command that libvirt used.

[root@polonium ~]# ps aux | grep libvirt | grep -v grep
  [ . . . ]
qemu       25641  0.9  0.9 5580140 646992 ?      Sl   04:45   0:27 /usr/bin/qemu-system-x86_64 -name guest=Debian-12-Guest-1,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-Debian-12-Guest-1/master-key.aes"} -machine pc-q35-8.1,usb=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu host,migratable=on -m size=4194304k -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296} -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 2d88e4ec-782f-4da6-961e-0266d08f845a -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=32,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device {"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"} -device {"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"} -device {"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"} -device {"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"} -device {"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"} -device {"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"} -device {"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"} -device {"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"} -device {"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"} -device {"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"} -device {"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"} -device {"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"} -device {"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"} -device {"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"} -device {"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/Debian-12-20231119.1.x86_64.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","driver":"qcow2","file":"libvirt-2-storage","backing":null} -device {"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1} -device {"driver":"ide-cd","bus":"ide.0","id":"sata0-0-0"} -netdev {"type":"tap","fd":"34","vhost":true,"vhostfd":"36","id":"hostnet0"} -device {"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:ec:db:35","bus":"pci.1","addr":"0x0"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev socket,id=charchannel0,fd=31,server=on,wait=off -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"} -device {"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"} -audiodev {"id":"audio1","driver":"none"} -vnc 127.0.0.1:1,audiodev=audio1 -device {"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"} -global ICH9-LPC.noreboot=off -watchdog-action reset -device {"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"} -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device {"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
[root@polonium ~]# 

Note: Command Summary

For cut and paste fiends, here's a list of all the commands run in this post.

# egrep '^flags.*(vmx|svm)' /proc/cpuinfo
dnf group install --with-optional virtualization
systemctl start libvirtd
systemctl enable libvirtd
# lsmod | grep kvm
# ip link show
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.2.0-amd64-netinst.iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS.sign
sha512sum -c SHA512SUMS --ignore-missing
gpg --keyserver keyring.debian.org --recv DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg --verify SHA512SUMS.sign SHA512SUMS

virt-install --name Debian-12-Guest-1 \
--description 'Debian 12 from netinst.iso' \
--ram 4096 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/Debian-12-20231119.1.x86_64.qcow2,size=20 \
--os-variant debian12 \
--network bridge=virbr0 \
--graphics vnc,listen=127.0.0.1,port=5901 \
--cdrom /var/lib/libvirt/images/iso/debian-12.2.0-amd64-netinst.iso \
--noautoconsole

virsh create Debian-12-Guest-1.xml
# virsh shutdown <virtual machine (name | id | uuid)>
#ps aux | grep qemu | grep -v grep

Future Steps

• Set up IPv4 NAT and IPv6/48 on the Node.
• Add IPv6/64 inside the guest.
• Set up Spice.
• Try other Linux and BSD guests.

Introduction

This article presents background information plus a quick and easy to follow recipe for installing, directly from Github source code repositories, the Python programming language and the Django web framework. The operating system used here is Fedora Rawhide. Also included is a note about using the install steps presented here on Debian 12.

As shown by the above screenshot, the quick, easy to follow, source-based Django install recipe given here actually works on Fedora Rawhide. Of course, the "without optimization" approach taken here is not intended for production. Instead, what we have here is a very fun introduction to building Python and Django from source code.

What Is Django?

Django is a web framework written in Python. The Django Project's landing page describes Django as "a high-level Python web framework that encourages rapid development and clean, pragmatic design."

Why Is Django Interesting?

A while back, I watched one of the CS50 lecture videos about Django. The CS50 video shows how simple and fast the Django web development process looks and how much Django must do in the background (for example, handling both database and web server functions). Web development using Django code seems cool -- very few lines of Django code are required to make dynamic web pages work!

Django Install Methods

At first sight, especially for someone coming without a Python background, installing Django seems really complicated! The Django Project's Quick install guide describes:

• installing Python,
• Optionally setting up a database,
• Three Django install methods
  • from an official Django release (recommended),
  • from your operating system's distribution,
  • from the latest development code, and
• Verifying the install.

The Quick install guide states that the first of the three install methods, installing from an official Django release, is "the best approach for most users."

However, for each of the three install methods, including for the recommended install method, the Django Quick install guide sends its readers to different sections of a significantly more detailed How to install Django page.

The more detailed How to install Django page explains that the recommended install from an official Django release involves both pip, the Package Installer for Python, and venv, the Python Virtual Environment Module.

Of course, for a Django install, Python itself, along with pip and venv might need to be installed if all three of these are not already present.

People use Django with a database. The database frequently is SQLite or PostgreSQL. Precompiled SQLite is installed during the preliminary Fedora development tools install discussed below. Although there is an official Github SQLite source mirror, SQLite uses fossil for source code management. Fossil seems so interesting and fun that it would make a great subject for another article. Thus, and even though PostgreSQL also is enticing, for the time being, on Fedora Rawhide, we can proceed with the distribution-installed SQLite.

Besides a database, Django also needs a web server. The official Django How to install tutorial recommends Apache web server with mod_wsgi. The web server install also might be another great subject for yet another LES Talk article. Meanwhile, as shown below, we can use Django's built-in Python web server. It is the built-in Python web server which served the page of which the above screenshot was taken.

Reading the Django Quick install guide was super helpful for important context, and reading the more detailed Django How to install guide was even more helpful for learning additional install details. But, the crux is, even after reading both guides, no specific install recipe was provided. Instead, readers of these well meant and helpful guides are in the end left facing a complex puzzle about how to choose among various install methods and sub-methods. Ironically, each of the install methods very kindly is intended to make installing Django easy!

Simplifying The Django Install

Having tried all three of the install methods listed in the official Django Quick install guide, and having tried several variations mentioned in the more detailed, official Django How to install guide, I wondered whether the Django install somehow could be easier. I wanted a quick Django install recipe!

Many source repositories have easy to find, basic configuration and compilation instructions. For example, Python's simple, standard build instructions are here. Often the simple, standard, "configure, make, make test, make install" series just works. Trying a source-based install seemed faster and easier, closer to upstream, and more fun than figuring out a specific install recipe based on one among the multiple, non-source-based install methods discussed in the official guides.

Source-based installs require a compiler and other development tools. In an ideal world, both the compiler and the development tools also themselves would be compiled from source by the end user. However, as shown here below, the Fedora distribution's package system can be used to quickly and easily install a compiler and other development tools.

Therefore, because of the simplicity of the standard build instructions and the ease of installing a compiler, I attempted, on Fedora Rawhide, a quick, source-based Django install. It seemed a good idea also to compile Python from C and to add pip and venv from their current development Python sources. That way, we get Django's friends built from their current sources to accompany our current, source-built Django. Of course, Fedora Rawhide runs on current Linux development kernels, so Rawhide might be an especially fun and good fit for current, source-built userland programs.

Clone, compile, test, and install times mentioned below are from an Intel E3-1275 v5 with 64 GB DDR4 ECC RAM and 2 x 512 GB NVMe disks rented from Hetzner for €26.70 per month (approximately US$28.34). Newer and faster processors with faster memory and faster disks will have shorter times.

Quick Source-Based Django Install Recipe

Part I: Install Development Tools On Fedora Rawhide

[not_oles@radium ~]$ sudo dnf groupinstall --with-optional "C Development Tools and Libraries"
[not_oles@radium ~]$ sudo dnf groupinstall --with-optional "Development Tools"
[not_oles@radium ~]$ sudo dnf groupinstall --with-optional "Development Libraries"

That's all, only these three commands. Probably they could be combined into a long, one line command, but I wanted to pause long enough between commands to read the individual output of each.

These three commands do not install Django, but they do install, respectively, 250, 585, and 186 development related software packages. That's 1,021 packages! Most of the newly installed packages are unneeded for our present purpose, but it was quick and easy to install everything. The entire one thousand plus package install took only about 8 minutes!

At least some Python must have been present in Fedora prior to the development tools install because dnf, the Fedora install tool, is itself Python based. Therefore, we need a trick to separate dnf installed programs from our own source-built programs. The PATH environment variable can help keep the dnf and source-built programs separated.

Part II: Check PATH And Enable Revert

• PATH

We need to check that both our root and our user PATH environment variables are set for our shell to find our newly installed /usr/local/bin binaries ahead of previously installed package system binaries. Fedora's PATH was okay because /usr/local/bin comes ahead of /usr/bin. Our install won't put anything in /usr/local/sbin, but for additional local installs from source, it's good for usr/local/sbin to come ahead of /usr/sbin.

[root@radium ~]# echo $PATH
/root/.local/bin:/root/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
[root@radium ~]# 

[not_oles@radium ~]$ echo $PATH
/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
[not_oles@radium ~]$

• Revert

Let's set up the ability to revert. We're going to be installing into /usr/local. So, in case we want to revert, we can use tar to back up /usr/local as it is prior to our install.

If we have enough disk space, there's no need for compressing our tar archive. Compression can take significant extra time.

tar is an old program! Old enough not to need the "-" before the flags!

[not_oles@radium ~]$ cd /usr
[not_oles@radium usr]$ sudo tar cvf local-revert.tar local

Part III: Install Python From Source

[not_oles@radium ~]$ cd /usr/local/src
[not_oles@radium src]$ time sudo git clone https://github.com/python/cpython.git
[not_oles@radium src]$ sudo mkdir cpython-obj
[not_oles@radium src]$ cd cpython-obj/
[not_oles@radium cpython-obj]$ time sudo ../cpython/configure --prefix="/usr/local" 
[not_oles@radium cpython-obj]$ time sudo make -j `nproc`
[not_oles@radium cpython-obj]$ time sudo make -j `nproc` test
[not_oles@radium cpython-obj]$ time sudo make -j `nproc` install

The above configure, make, make test, and make install procedure installs pip, as seen in the following snippet from the make install output.

Successfully installed pip-23.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour 
with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

The newly compiled and installed items are:

[not_oles@radium ~]$ ls -l /usr/local/bin
total 28408
lrwxrwxrwx. 1 root root        8 Oct 15 22:09 idle3 -> idle3.13
-rwxr-xr-x. 1 root root      100 Oct 15 22:08 idle3.13
-rwxr-xr-x. 1 root root      230 Oct 15 17:40 pip3
-rwxr-xr-x. 1 root root      230 Oct 15 17:40 pip3.13
lrwxrwxrwx. 1 root root        9 Oct 15 22:09 pydoc3 -> pydoc3.13
-rwxr-xr-x. 1 root root       85 Oct 15 22:08 pydoc3.13
lrwxrwxrwx. 1 root root       10 Oct 15 22:09 python3 -> python3.13
-rwxr-xr-x. 1 root root 29065592 Oct 15 22:08 python3.13
-rwxr-xr-x. 1 root root     3026 Oct 15 22:08 python3.13-config
lrwxrwxrwx. 1 root root       17 Oct 15 22:09 python3-config -> python3.13-config
[not_oles@radium ~]$ 

Note that the pip described in the warning as successfully installed was installed as pip3 and as pip3.13, but not as plain pip. For Python itself, similar to pip, pip3, and pip3.13, the above configure, make, make test, and make install procedure installed python3.13 and python3 as a symbolic link to python3.13, but not python. sqlite3 also is not installed.

On Fedora Rawhide, the complete clone, configure, make, make test, and make install Python procedure took 9 minutes and 51 seconds.

Part IV: Install nox From Source

nox is a requirement for installing pip from source. The pip development Getting Started page says to install nox inside a virtual environment. However, the nox Github readme.md says, "Nox is designed to be installed globally (not in a project virtual environment)." On Fedora Rawhide, both types of installs seem initially to work. Here is the global nox install.

[not_oles@radium ~]$ cd /usr/local/src
[not_oles@radium src]$ time sudo git clone https://github.com/wntrblm/nox.git
[not_oles@radium src]$ time sudo cd nox
[not_oles@radium nox]$ time sudo python3 -m pip install -e . # Note final posix dot.

Part V: Create, Activate, And Enter A Python Virtual Environment

[not_oles@radium ~]$ time python3 -m venv .venv
[not_oles@radium ~]$ source .venv/bin/activate
(.venv) [not_oles@radium ~]$ cd .venv
(.venv) [not_oles@radium .venv]$

Part VI: Install pip From Source In The Virtual Environment

# Reference:  https://pip.pypa.io/en/latest/development/getting-started/
(.venv) [not_oles@radium .venv]$ time git clone https://github.com/pypa/pip
(.venv) [not_oles@radium .venv]$ cd pip
(.venv) [not_oles@radium pip]$ time python3 -m pip install -e . # Note final posix dot.

Part VII: Install Django From Source In The Virtual Environment

# Reference: https://docs.djangoproject.com/en/4.2/topics/install/#installing-development-version
(.venv) [not_oles@radium .venv]$ time git clone https://github.com/django/django.git
(.venv) [not_oles@radium .venv]$ cd django/
(.venv) [not_oles@radium django]$ time python3 -m pip install -e .

Part VIII: How To Exit The Virtual Environment

(.venv) [not_oles@radium django]$ deactivate
[not_oles@radium django]$ cd
[not_oles@radium ~]$ 

Part IX: Test

• Normal Environment

[not_oles@radium ~]$ which python
/usr/bin/python
[not_oles@radium ~]$ /usr/bin/python -V
Python 3.12.0
[not_oles@radium ~]$ which python3
/usr/local/bin/python3
[not_oles@radium ~]$ /usr/local/bin/python3 -V
Python 3.13.0a1+
[not_oles@radium ~]$ which nox
/usr/local/bin/nox
[not_oles@radium ~]$ /usr/local/bin/nox --version
2023.4.22
[not_oles@radium ~]$ which pip
/usr/bin/pip
[not_oles@radium ~]$ /usr/bin/pip -V
pip 23.2.1 from /usr/lib/python3.12/site-packages/pip (python 3.12)
[not_oles@radium ~]$ which pip3
/usr/local/bin/pip3
[not_oles@radium ~]$ /usr/local/bin/pip3 -V
pip 23.2.1 from /usr/local/lib/python3.13/site-packages/pip (python 3.13
[not_oles@radium ~]$ which django-admin # PATH in global environment
/usr/bin/which: no django-admin in (/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
[not_oles@radium ~]$ which sqlite
/usr/bin/which: no sqlite in (/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
[not_oles@radium ~]$ which sqlite3
/usr/bin/sqlite3
[not_oles@radium ~]$ sqlite3 -version
3.43.1 2023-09-11 12:01:27 2d3a40c05c49e1a49264912b1a05bc2143ac0e7c3df588276ce80a4cbc9balt1 (64-bit)
[not_oles@radium ~]$ uname -r
6.6.0-0.rc5.20231013git10a6e5feccb8.44.fc40.x86_64
[not_oles@radium ~]$ cat /etc/fedora-release 
Fedora release 40 (Rawhide)
[not_oles@radium ~]$ 

• Virtual Environment

(.venv) [not_oles@radium .venv]$ which python
~/.venv/bin/python
(.venv) [not_oles@radium .venv]$ python -V
Python 3.13.0a1+
(.venv) [not_oles@radium .venv]$ which python3
~/.venv/bin/python3
(.venv) [not_oles@radium .venv]$ python3 -V
Python 3.13.0a1+
(.venv) [not_oles@radium .venv]$ which nox
/usr/local/bin/nox
(.venv) [not_oles@radium .venv]$ nox --version
2023.4.22
(.venv) [not_oles@radium .venv]$ which pip
~/.venv/bin/pip
(.venv) [not_oles@radium .venv]$ pip --version
pip 24.0.dev0 from /home/not_oles/.venv/pip/src/pip (python 3.13)
(.venv) [not_oles@radium .venv]$ which pip3
~/.venv/bin/pip3
(.venv) [not_oles@radium .venv]$ pip3 --version
pip 24.0.dev0 from /home/not_oles/.venv/pip/src/pip (python 3.13)
(.venv) [not_oles@radium .venv]$ which django-admin
~/.venv/bin/django-admin
(.venv) [not_oles@radium .venv]$ django-admin --version
5.1.dev20231015082711
(.venv) [not_oles@radium .venv]$ which sqlite # Get PATH inside virtual environment :)
/usr/bin/which: no sqlite in (/home/not_oles/.venv/bin:/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
(.venv) [not_oles@radium .venv]$ which sqlite3
/usr/bin/sqlite3
(.venv) [not_oles@radium .venv]$ sqlite3 -version
3.43.1 2023-09-11 12:01:27 2d3a40c05c49e1a49264912b1a05bc2143ac0e7c3df588276ce80a4cbc9balt1 (64-bit)
(.venv) [not_oles@radium .venv]$ uname -r
6.6.0-0.rc5.20231013git10a6e5feccb8.44.fc40.x86_64
(.venv) [not_oles@radium .venv]$ cat /etc/fedora-release 
Fedora release 40 (Rawhide)
(.venv) [not_oles@radium .venv]$ 

• Web Server

We can launch a web server, and then our browser can see the Django default html index page shown in the screenshot above!

[not_oles@radium ~]$ cd .venv
[not_oles@radium .venv]$ source bin/activate
(.venv) [not_oles@radium .venv]$ time django-admin startproject mysite
(.venv) [not_oles@radium .venv]$ cd mysite/
(.venv) [not_oles@radium mysite]$ python3 manage.py runserver 127.0.0.1:42365

Time Summary

Debian 12

Newly installed out of its netinstall.iso box, Debian 12 Linux seems to use /usr/local/lib for apt Python installations. In other words, /usr/local/lib comes with subdirectories /python3.11 and /python3.11/dist-packages preinstalled.

Moreover, Debian valiantly guards /usr/local. The above CPython "configure, compile, test, install" command sequence does not install any pip. Also, on Debian 12, the above nox install command failed. There was a helpful error message, which explained, "This environment" [that is, I guess, global /usr/local] "is externally managed. . . ."

Discussion

• What are the advantages and disadvantages of installing nox globally versus installing nox inside a virtual environment?
• Is using python for the packaged install and python3 for the compiled install a good idea?
• Should we do something like sudo ln -s /usr/local/bin/python3.13 /usr/local/bin/python?
• CPython has a helpful Setup and Building Guide. In addition to configuring and compiling, git, dependencies, several operating systems, and troubleshooting also are discussed. The Guide says, "There is normally no need to install your built copy of Python! The interpreter will realize where it is being run from and thus use the files found in the working copy."
• I need to study up on the cpython configure options, including --with-pydebug and --enable-optimizations.
• The Cpython test_dtrace formerly succeeded on Rawhide, but it has failed since about the time earlier this month when Python 3.12 was released. There is an old bug report from 2019 which has similar log messages. Configuring Cpython with --with-dtrace did not help.

Also note that here I'm downloading a fp16 model and converting it to q8 GGUF. In practice you can skip over those steps and just download ready made quantized GGUF models from TheBloke's huggingface repo.. i.e. You'd modify the download model step to point to a quantized GGUF model and skip the generate and quantize step after that.

This assumes Ubuntu 22.04 - you may need to do stuff like install python3 if you're on a different distro

Check that we have a GPU

apt update && apt upgrade
apt install hwinfo -y
hwinfo --gfxcard --short

Set up nvidia driver and SDK

apt install nvidia-driver-535-server nvidia-dkms-535-server nvidia-cuda-toolkit -y
reboot
nvidia-smi
nvcc --version

Grab llama.cpp and build it

git clone https://github.com/ggerganov/llama.cpp
apt install cmake -y
cd llama.cpp
mkdir build
cd build
cmake .. -DLLAMA_CUBLAS=ON
cmake --build . --config Release
cd ..

Download a model

mkdir -p /root/llama.cpp/models/llama2-fp16
python3 -m pip install huggingface_hub
python3
from huggingface_hub import snapshot_download
snapshot_download(repo_id="TheBloke/Llama-2-13B-Chat-fp16", revision="main",local_dir="/root/llama.cpp/models/llama2-fp16/")
quit()

Generate GGUF file

python3 -m pip install gguf sentencepiece
python3 convert.py ./models/llama2-fp16/

Quantize it

cd ./build/bin
./quantize ../../models/llama2-fp16/ggml-model-f16.gguf ../../models/llama2-q8.gguf q8_0

Run it

./main -m ../../models/llama2-q8.gguf -ngl 99 --color -p "Tell me a story about a unicorn!"

Tell me a story about a unicorn!

Once upon a time, in a far-off land of rolling hills and sparkling streams, there lived a beautiful unicorn named Luna. She had a shimmering coat of silver and white, and her horn was as bright as the stars in the night sky.

Luna lived a peaceful life, roaming the forests and meadows, and making friends with all the creatures she met. She loved to play with the butterflies and dance with the flowers, and she could make the most beautiful music with her horn.

One day, a wicked witch cast a spell on the land, causing all the plants and animals to become sick and tired. The unicorns were especially affected, and their beautiful coats became dull and lifeless.

Luna knew that she had to do something to save her friends and the land they lived in. She set out on a journey to find the witch and break her spell.

As she traveled through the forest, Luna met many creatures who were suffering from the witch's spell. She used her horn to heal them and bring them back to life. She also met a brave knight who had been searching for the witch for many years. Together, they journeyed on, determined to defeat the wicked witch and bring peace back to the land.

Finally, after many days of traveling, they came to the witch's castle. It was a dark and gloomy place, surrounded by a moat of swirling black water. But Luna was not afraid. She knew that her horn could break any spell, no matter how powerful.

She and the knight entered the castle, ready to face whatever dangers lay inside. As they made their way deeper into the castle, they came across the witch herself. She was a terrifying sight, with warts and a crooked nose, and a cackle that sent chills down your spine.

But Luna was not afraid. She raised her horn and pointed it at the witch, ready to break the spell. The witch laughed and tried to stop her, but Luna's horn was too powerful. With one blast of magic, the spell was broken, and the land was once again filled with light and life.

The creatures who had been turned to stone were returned to their true forms, and they cheered and celebrated as Luna and the knight emerged from the castle. The witch was banished from the land forever, and peace was restored.

And Luna, the little unicorn with the powerful horn, lived happily ever after, knowing that she had saved her homeland from the evil witch's spell. The end.

For over four years now I have been enjoying a Google Cloud Platform (GCP) Free Tier Debian VPS. During this time, Debian's package system cache has grown to use more than half of the VPS's 10 GB of file storage space to cache .deb package files. This post explains the history and the use of this wonderful, free VPS (Thanks, Google!); shows the size of the package cache; and explains how to reclaim the large amount of disk space used for the cache.

VPS Creation in 2019

According to the Google Cloud Console, my e2-micro instance was created on January 27, 2019. That was more than four and a half years ago!

Larger File System Size Possible Within Current GCP Free Tier Limits

Looking at the current GCP Free Tier Resource Limits suggests that the Compute Engine limit is "30 GB-months standard persistent disk." My VPS is only 10 GB, so maybe, when I need it, I could increase the file system size to add an additional 20 GB.

VPS Use

This GCP VPS has been in modest, but continuous, steady use throughout more than four years. It serves my tiny personal website, holds a few small backup files, and runs TinyProxy. Tinyproxy works great for accessing certain websites in the US when I am in Mexico.

I log in to the VPS frequently. I wouldn't say every week for sure, but, if I am working on something, I log in many times in one week. Of course, to update the VPS's Debian operating system, I regularly run apt-get update and then apt-get upgrade.

Running EOL Debian 10 Buster

My GCP VPS is still running Debian 10 Buster (End of Life (EOL) a year ago, September 10, 2022), which might be okay, because the check-support-status utility doesn't list any installed packages without Long Term Support (LTS).

root@gc:~# man check-support-status
root@gc:~# check-support-status
root@gc:~# echo $?
0
root@gc:~# 

Great VPS Uptime!

Usually I run the w command when I log in. Here's what I saw today. 590 days of uptime isn't bad!

root@gc:~# w
 20:43:01 up 590 days, 16:55,  1 user,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
tom      pts/0    187.189.238.2    19:30    0.00s  0.08s  0.01s sshd: tom [priv]    
root@gc:~# 

Low Free Disk Space

One thing I have noticed is that free disk space on the VPS has been decreasing. Here is what free disk space looked like earlier today.

root@gc:~# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       9.8G  8.7G  673M  93% /
root@gc:~# 

The VPS image was set up without a swap partition. Since I might want to add more website or backup files, install additional utilities, or even add a swap file, I thought the remaining 673 MB might be a little small. I decided to take a look around and see if I could find a place or a few places where I could remove a big file that no longer was needed.

root@gc:/# du -sh /home/tom /var/www
77M     /home/tom
67M     /var/www
root@gc:/# 

Hmm! It didn't look like my home directory or even my website files were too piggy! Looking some more, I was surprised to find that /var was using a lot of space!

root@gc:/# du -sh /var
5.9G    /var
root@gc:/# 

Wow! "Lots of logs?" I wondered.

root@gc:/# du -sh /var/*
1.2M    /var/backups
5.7G    /var/cache
156M    /var/lib
4.0K    /var/local
0       /var/lock
13M     /var/log
3.3M    /var/mail
4.0K    /var/opt
0       /var/run
8.7M    /var/spool
12K     /var/tmp
67M     /var/www
root@gc:/# 

What's using all that space in /var/cache?

root@gc:/# du -sh /var/cache/*
108K    /var/cache/apparmor
5.7G    /var/cache/apt
3.8M    /var/cache/debconf
20K     /var/cache/ldconfig
2.5M    /var/cache/man
4.0K    /var/cache/private
root@gc:/# 

Oh! It's the package system! Four years of .deb files live there! 5.7 GB! Looking at the full output of the below command without the pipe to wc, it was instantly clear that all except the first five of the 346 output lines were .deb files. That's 341 .deb files! Together, the 341 cached .deb files used more than half of my VPS's disk space!

root@gc:/# ls -l /var/cache/apt/* | wc
    346    3090   28173
root@gc:/# 

About .deb Files

Binary .deb files are packages containing binaries which are "directly useable by dpkg." Source .deb files contain source code.

Luckily for small VPSes, after the system has been updated, the .deb files may be removed.

How To Reclaim Disk Space

From man apt-get:

       clean
           clean clears out the local repository of retrieved package files. It removes
           everything but the lock file from /var/cache/apt/archives/ and
           /var/cache/apt/archives/partial/.

After running apt-get clean the VPS got its empty disk space back! 6.4 GB available instead of 673 MB!

root@gc:~# apt-get clean
root@gc:~# ls -l /var/cache/apt/*
total 4
-rw-r----- 1 root root    0 Jan 24  2019 lock
drwx------ 2 _apt root 4096 Sep 13 21:22 partial
root@gc:~# du -sh /var/cache/apt
64K     /var/cache/apt
root@gc:~# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       9.8G  3.0G  6.4G  32% /
root@gc:~# 

Disabling The Cache

One possible way to prevent running out of disk space might be to disable the caching of .deb files. From man 5 apt-conf:

DIRECTORIES

       [ . . . ]

       Dir::Cache contains locations pertaining to local cache information, such as the two
       package caches srcpkgcache and pkgcache as well as the location to place downloaded
       archives, Dir::Cache::archives. Generation of caches can be turned off by setting
       pkgcache or srcpkgcache to "".

Please see also this old Bug #753531 from 2014, which was very fun to read because it almost said rm -rf /*!

It seems possibly safer and generally less work all around to leave the .deb caching defaults as they are. It's easy to empty the cache once in awhile with apt-get clean.

Conclusion

For small Debian VPSes which have been in use a long time, you might want to run apt-get clean to reclaim the large amount of disk space used by the apt cache.

0. Advance Notice

• Special thanks to @Not_Oles !
• This tutorial has been verified to work with Debian 11 on Hetzner's dedicated server.
• All commands are executed as root.

• In this tutorial the host's network configuration is:
  Network Interface: eno1
IPv4 Address: 192.0.2.2/26
IPv4 Gateway: 192.0.2.1
IPv6 Address: 2001:db8:ace:babe::1/64
IPv6 Gateway: fe80::1

• We want to create a private network 192.168.0.0/24 for VMs; we want to use 192.168.0.1 as the IPv4 gateway and 2001:db8:ace:babe::1 as the IPv6 gateway for VMs.

• We want to create a VM whose IPv4 address is 192.168.0.2 and IPv6 address is 2001:db8:ace:babe:cafe::1/80.

1. Enable IP Forwarding

Open /etc/sysctl.conf. Find net.ipv4.ip_forward; uncomment this line and set the value to 1. Do the same for net.ipv6.conf.all.forwarding.

Save the file; then run sysctl -p to apply changes.

2. Install Required Packages

apt install qemu-system qemu-utils libvirt-clients libvirt-daemon-system virtinst bridge-utils

3. Modify the Network Configuration

3.1 Make a copy of the original /etc/network/interfaces

cp -p /etc/network/interfaces /etc/network/interfaces.backup

3.2 Open /etc/network/interfaces and edit

Comment out eno1's IPv6 configuration, then add the following line:

iface eno1 inet6 manual

Continue to add the following lines to create an interface br0 for KVM networking:

auto br0
iface br0 inet static
    address 192.168.0.1/24
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
iface br0 inet6 static
    address 2001:db8:ace:babe::1/64
    up ip -6 route add default via fe80::1 dev eno1

bridge ports none means br0 is not attached to any physical interface. bridge_stp off disables Spanning Tree Protocol; usually we don't need it in simple cases. bridge_fd 0 sets the forwarding delay time to 0; 0 is good in simple cases. The iptables line allows LAN nodes with private IP addresses to communicate with external public networks. The ip -6 route line specifies the IPv6 gateway.

Save the file; then run systemctl restart networking.service to apply changes.

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eno1
iface eno1 inet static
    address 192.0.2.2
    netmask 255.255.255.192
    gateway 192.0.2.1
    # route 192.0.2.0/26 via 192.0.2.1
    up route add -net 192.0.2.0 netmask 255.255.255.192 gw 192.0.2.1 dev eno1
iface eno1 inet6 static
    address 2001:db8:ace:babe::1
    netmask 64
    gateway fe80::1

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eno1
iface eno1 inet static
    address 192.0.2.2
    netmask 255.255.255.192
    gateway 192.0.2.1
    # route 192.0.2.0/26 via 192.0.2.1
    up route add -net 192.0.2.0 netmask 255.255.255.192 gw 192.0.2.1 dev eno1
iface eno1 inet6 manual

auto br0
iface br0 inet static
    address 192.168.0.1/24
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
iface br0 inet6 static
    address 2001:db8:ace:babe::1/64
    up ip -6 route add default via fe80::1 dev eno1

4. Create the VM

virt-install --name YOUR_VM_NAME --ram MEMORY_SIZE_IN_MB --vcpus=NUMBER_OF_CORES --disk /PATH/TO/VIRTUAL/DISK/IMAGE.qcow2,device=disk,bus=virtio,size=DISK_SIZE_IN_GB,format=qcow2 --graphics vnc,listen=0.0.0.0,port=VNC_PORT,password=VNC_PASSWORD --network bridge=br0 --noautoconsole --cdrom /PATH/TO/ISOFILE.iso --boot cdrom,hd

5. Configure Guest Networking

Because we haven't configured the DHCP service on the host, we have to manually set the VM network settings. If you are using a netinst image to install the OS, then you need to set the IP address to 192.168.0.2/24 and the gateway to 192.168.0.1 during the installation.

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-hotplug ens3
iface ens3 inet static
    address 192.168.0.2/24
    gateway 192.168.0.1
iface ens3 inet6 static
    address 2001:db8:ace:babe:cafe::1/80
    dns-nameservers 2606:4700:4700::1001
    gateway 2001:db8:ace:babe::1

In part one of this guide we will set up an outbound only mail relay server that will connect and send mail via MXRoute.
In part two we will show how to set up sendmail on any VPS that you want to relay mail through the outbound only mail relay server that you set up in part one. You can also use postfix or any other mail service on the secondary servers to relay through the outbound only mail relay server.

NOTE: In an effort to make this rather long guide a more reasonable size I have put many of the code blocks that you will need to copy into display pull downs. You will need to click the to see this information.

Part One

Install required packages

CentOS 7:
yum -y install sendmail sendmail-cf cyrus-sasl cyrus-sasl-plain
yum -y remove postfix

CentOS 8, Alma8linux, Rockylinux:
dnf -y install sendmail sendmail-cf cyrus-sasl cyrus-sasl-plain
dnf -y remove postfix

Make sure your hostname is set

Edit /etc/hosts and ensure there is a line [your IPv4] tab [your host name]
Edit /etc/sysconfig/network and ensure there is a line HOSTNAME=[your host name]

Setup the certificates

Get a cert from Let's Encrypt for your mail relay server (I am not showing how to do this here)
Make a subdirectory called "certs" under /etc/mail/:
mkdir /etc/mail/certs

While in the directory /etc/mail/certs execute the commands:
openssl dhparam -out dhparam.pem 4096
and
ln -s /etc/ssl/certs/ca-bundle.crt ca-bundle.crt
and
chmod 600 dhparam.pem sendmail.pem

You should now have three files in your /etc/mail/certs directory

ca-bundle.crt [a soft link]
dhparam.pem
sendmail.pem

The certificate will need to be undated before it expires.
I should add a script to auto update the mail server cert from LetsEncrypt. (coming soon™)

Configure sendmail.mc

Move the original sendmail.mc to sendmail.mc.org:
mv /etc/mail/sendmail.mc /etc/mail/sendmail.mc.org

After making the above changes execute the command:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Setup MXRoute authinfo

After making the above changes ensure that you are in the directory /etc/mail and execute the command:
makemap hash authinfo < authinfo

Enable sendmail, saslauthd, and reboot

Execute the following commands to enable sendmail and saslauthd to start on boot and then reboot:
/usr/bin/systemctl enable sendmail.service
/usr/bin/systemctl enable saslauthd.service
reboot

Set SPF DNS record

Add the IPv4 address of this mail relay server and MXRoute to your DNS spf record:
TXT "v=spf1 +ip4:[IPv4 address of this mail relay server] include:mxlogin.com -all"

Is it working ?

Test if everything works as expected by running this command:
echo "Subject: hello" | sendmail you@yourdomain
(replace you@yourdomain with your email address)

Check the log file:
cat /var/log/maillog

If you see stat=Sent (OK, you are sending your mail through MXRoute:

Jan 9 16:40:55 f sendmail[2246]: 309KF3K5001390: to=<you@yourdomain>, ctladdr=<root@[your relay mail server]> (0/0), delay=01:25:51, xdelay=00:00:01, mailer=relay, pri=750282, relay=friday.mxlogin.com. [159.69.65.104], dsn=2.0.0, stat=Sent (OK id=1pEzt1-0004Nf-UP)

If you see stat=Deferred, it is not working:

Jan 9 15:30:02 f sendmail[1848]: 309KF3K5001390: to=<you@yourdomain>, ctladdr=<root@[your relay mail server]> (0/0), delay=00:14:58, xdelay=00:00:00, mailer=relay, pri=390282, relay=friday.mxlogin.com., dsn=4.0.0, stat=Deferred

NOTES

If you make any revisions to /etc/mail/sendmail.mc you will need to execute:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

If you make any revisions to /etc/mail/authinfo you will need to execute:
makemap hash authinfo < authinfo

Then restart sendmail:
/usr/bin/systemctl restart sendmail.service

So that the changes you made will take effect.

Are we done yet ?

If you are NOT planning to have any other VMs relay through this mail relay server you are done. All mail from this VM will relay via MXRoute. While sendmail is listening on port 2525 no mail will be relayed except from localhost.

If you DO want to relay other VMs mail through this mail server to also send their mail via MXRoute then continue below.

Allow other VMs to relay through this server

Open /etc/mail/access in your favorite text editor and add each of the VMs that you want to allow mail to relay through this server by adding their IP address, each on a seperate line, at the bottom of the file as Connect:123.123.123.123 RELAY.

After you have added all the VMs you wish to allow to relay to MXRoute via this mail server, and saved the file, execute this command:
makemap hash /etc/mail/access.db < /etc/mail/access

Restart sendmail:
/usr/bin/systemctl restart sendmail.service

You will need to execute the two commands above anytime you make any additions or subtractions to the /etc/mail/access file for the changes to take effect.

Don't forget to open your firewall

To relay mail via your outbound only mail server from other VMs you must open your firewall for port 2525 access.

For firewalld:
firewall-cmd --permanent --zone=public --add-port=2525/tcp
service firewalld stop
service firewalld start

Part Two

In part two we will show how to set up sendmail on any VPS that you want to relay mail through the mail relay server that you set up in part one. It's much easier and faster.

Install required packages

CentOS 7:
yum -y install sendmail sendmail-cf
yum -y remove postfix

CentOS 8, Alma8linux, Rockylinux:
dnf -y install sendmail sendmail-cf
dnf -y remove postfix

Make sure your hostname is set

Edit /etc/hosts and ensure there is a line [your IPv4] tab [your host name]
Edit /etc/sysconfig/network and ensure there is a line HOSTNAME=[your host name]

Edit the sendmail.mc file

Open /etc/mail/sendmail.mc with your favorite text editor and add the lines in the pull down near the bottom of the file between the quoted lines shown above and below the pull down box.

dnl MASQUERADE_DOMAIN(mydomain.lan)dnl

MAILER(smtp)dnl
MAILER(procmail)dnl
dnl MAILER(cyrusv2)dnl

After copying the code to /etc/mail/sendmail.mc you will need to modify the following:
1. Change 123.123.123.123 to the IPv4 of the server that is being installed.
2. Change myrelay.mydomain to the hostname.domainname of your mail relay server from part one.

After making the above changes execute the command:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Enable sendmail and reboot

Execute the following commands to enable sendmail to start on boot and start now :
/usr/bin/systemctl enable sendmail.service
reboot

Set SPF DNS record

If you have not done so previously, add the IPv4 address of the mail relay server and MXRoute to your DNS spf record of the domains for which this VM will be sending mail:
TXT "v=spf1 +ip4:[IPv4 address of mail relay server] include:mxlogin.com -all"

Is it working ?

Test if everything works as expected by running this command:
echo "Subject: hello" | sendmail you@yourdomain
(replace you@yourdomain with your email address)

Check the log file:
cat /var/log/maillog

If you see stat=Sent, and Message accepted for delivery you are sending your mail through your mail relay server:

Jan 10 04:27:12 localhost sendmail[2993586]: STARTTLS=client, relay=my.relayserver., version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Jan 10 04:27:13 localhost sendmail[2993586]: 30A9RCpK2993585: to=you@yourdomain, ctladdr=<root@localhost> (0/0), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=149367, relay=my.relayserver. [IPv4 for my.relayserver], dsn=2.0.0, stat=Sent (30A9RCkP001947 Message accepted for delivery)

On the mail relay server from part one these are the logs of the same email message successfully being relayed through to MXRoute:

Jan 10 04:27:13 my sendmail[1947]: 30A9RCkP001947: from=<root@the1stvps>, size=11346, class=-60, nrcpts=1, msgid=<202301100927.30A9R2jw2993246@the1stvps>, proto=ESMTPS, daemon=MTARelay, relay=the1stvps [IPv4 of 1st vps]
Jan 10 04:27:16 my sendmail[1949]: STARTTLS=client, relay=friday.mxlogin.com., version=TLSv1.2, verify=OK, cipher=ECDHE-RSA-AES128-GCM-SHA256, bits=128/128
Jan 10 04:27:17 my sendmail[1949]: 30A9RCkP001947: to=<you@yourdomain>, delay=00:00:04, xdelay=00:00:04, mailer=relay, pri=239346, relay=friday.mxlogin.com. [159.69.65.104], dsn=2.0.0, stat=Sent (OK id=1pFAuc-0004BM-Uh)

If you see stat=Deferred Connection timed out , it is not working.
In this case the firewall was not open for port 2525 on mail relay server:

Jan 10 04:24:01 localhost sendmail[2987677]: 30A8S7uH2987677: to=<you@yourdomain>, delay=08:45:57, xdelay=00:00:00, mailer=relay, pri=931045, relay=my.mailrelayserver., dsn=4.0.0, stat=Deferred:Connection timed out with my.mailrelayserver.

If you see stat=User unknown , it is not working.
In this case the IPv4 of this server was not in the /etc/mail/access file on the mail relay server:

Jan 10 22:30:47 localhost sendmail[3093806]: 30B3UkCR3093806: to=you@yourdomain, delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=32612, relay=my.relayserver. [IPv4 of my.relaysever], dsn=5.7.1, stat=User unknown

This will be shown on the mail relay server as the following Relaying denied error.

Jan 10 22:30:47 my sendmail[1363]: 30B3UjUR001363: ruleset=check_rcpt, arg1=<you@yourdomain>, relay=the1stvps [IPv4 of 1st VPS], reject=550 5.7.1 <you@yourdomain>... Relaying denied

Are we done yet ?

Yes, for this guide that is it for now.....
But no you are not done yet. At a minumum servers that send outbound mail should use DKIM in addition to having SPF records and all the good stuff that MXRoute provides.

A warning from MXRoute so as to not cause problems

@jarland said:
It’s worth noting that a slight variation on this kind of setup consumes a significant amount of my time trying to protect the infrastructure from being overwhelmed by pure junk that drags down IP reputation by increasing the association between our IPs and spam folders. These are the big line items:

1. Almost no one who does this cares that their hostname isn’t an actual valid domain, forcing me to locate all of their server hostnames and add them to a banned sender list so they get errors instead of flooding the queue with junk I can’t deliver or bounce.
2. That cron job you have set to run every minute sends a notification, and the person who sets this up commonly has all of these emails sent to their Gmail address.
3. A simple command line interface or a cron job notification lacks the required headers that a legitimate email client would add without the user noticing.

Number 1 causes our mail queues to be packed full of “root@vps{1..999}” emails that I can’t and won’t deliver because they’re not from valid envelope senders. But I can’t say that these are inherently bad and prevent them from going out before I identify each one of them because temporary DNS issues could cause a sending address to appear invalid that isn’t later. So I queue them, but then I audit the queues every day for this junk and use it to build my banned sender list (ex. *@localhost, *@vps1, *@vps.local, etc.).

Number 2 loops in with #1, and the two items work hand in hand to try to overly associate our IP addresses with Gmail’s spam folder, and number 3 comes in here as part of that. A bad sending address, a cron job notification that the user never meant for human consumption (and will never read, save for the 1 in a million user who might), and a lack of valid headers combined with Gmail’s threading feature mean you see one little junk email in your Spam folder, but you actually sent 1,440 junk emails per cron job to Gmail that were all destined to the spam folder. Combine that with the 600 other users doing the same thing, and you then have 864,000 emails from our platform daily, which are guaranteed to land in Gmail’s spam folder. Almost none of those users even meant to send those emails. The users rarely have any idea that they sent them, and they have no plan of gaining anything from it. It’s hard to let all of that through and then tell people I’m focused on doing everything I can to ensure inbox delivery.

All this to say, it’s good to be able to configure your systems to do this, but please keep one thing in mind: The MXroute platform is designed to send emails that are intended for either human consumption or automatic processing. It is not intended to be used as a garbage delivery system for Gmail. It also can’t be used as a garbage delivery system for Gmail, expecting that intentionally sent emails have a high probability of landing in the inbox. It can be one or the other. It can’t be both. Doing everything I can to get intentional emails into the inbox has always been my top focus. So please, if you do this, take a moment to consider what your server is doing and how it may use this configuration in ways you weren’t thinking of at the time.

Do make sure that your sending addresses are always valid. If your hostname isn’t valid, has no MX or SPF records itself, then your cron jobs and other system emails are going to be sending as invalid envelope senders. Think “root@localhost” that’s not valid, even “root@myvps.mydomain.tld” is invalid if myvps.mydomain.tld lacks valid MX/SPF. Most of that is fine if you’re sending to yourself to be delivered to an inbox on the same MXroute server that your service is hosted on, but if you’re sending these out to remote recipients that’s when you really need to dot your I’s and cross your T’s, making sure you have everything done properly.

-By A. Vyas, January 2023

Read Part I here: Five Ways to Keep Your Hosting Costs Under Control - Part I

Introduction

This is the second and concluding part of a two part series on how you can reduce your web hosting costs in times of high inflation and increasing web hosting fees. In the previous part, we saw how a person "Lata" was able to save nearly 30 US dollars a month by adopting three different approaches to manage her web hosting plans.

To summarise, the three approaches she adopted so far were:
I. Consolidate the hosting plans
II. Moving the hosting to a cheaper location(s)
III. Lower the specifications a notch or two

In this part, let us explore some more ways that she followed, and determine how much savings she was able to achieve in practical terms.

IV. Check those idlers and duplicates

Idlers are the bane of the Low End aficionados. Lata's case was no different. She had multiple idlers in her webhosting "Empire", and quite a collection of hosting services were in Singapore. From Lata's home location, Singapore offered the greatest set of options and a reasonable ping ~ 45 ms. It was not the fastest or the best latency she could get, but it was much better than locations in Western Europe (~ 150 ms) or the East Coast, USA (~ 220 ms).

Over the past couple of years Lata kept subscribing to newer plans in Singapore, it almost became a habit of sorts. Recently, she decided to cut the number of plans to half. Gone was a small server, a larger KVM, and a reseller hosting plan. All three were mostly idling.

Lata also had multiple KVM plans in Western Europe. Some could be transferred out, but the renewal for these plans was only a few weeks away. She set them to be terminated at the conclusion of their respective periods.

Scheduling to terminate the service at the end of the billing period also gave her an opportunity to backup and migrate her data in a planned manner, instead of rushing things at the last minute.

Tip: You can consider transferring out the plans out to other users in the LE community. However, the level of effort and transfer fees in many cases may outweigh the savings.

Estimated savings till March 2023 : about 30 US Dollars

Tip: Do you have a few extra idlers that are higher than your comfort level? Do you see an opportunity to cut down on a few plans? A great place to start would be to check if you have multiple plans in the same location.

V. Check the pricing terms, mode of payment, and currency

Lata, who is based in Bengaluru in India, was paying for most services in US Dollars through PayPal. She began to explore providers who accepted payment in Indian Rupees (INR). The reason for preferring PayPal will be highlighted in a different post, since it involves a deep dive into regulation, tax implications, as well as other technicalities.

She switched some of her services to providers who accepted payment in INR, and paid them using services such as Razorpay. Paying for longer terms (6 months or higher) helped her and the web host save some percentage points in processing fees as well.

The downside of this approach is that sometimes, this method may not be practical.

For example, Lata had VPS plan with one host from India, for a VPS located in Mumbai. Their monthly charges were 4 US dollars. For annual payment, they offered a 8 dollar discount, that is 40 US Dollars per year. However, Lata was required to pay in US Dollars.

Typically, INR: US Dollar exchange rate is between 70 and 72 (i.e. 70 INR = 1 US Dollar). In recent months, it is hovering closer to INR 80-83 range. Adding Paypal fees, etc- the rate climbs to 85 or 86 INR per US Dollar. In other words, Paypal adds an overhead that negates any potential cost savings. In Lata's case, it resulted in savings of only 2 US Dollars. She dropped the idea of switching the currency and payment terms altogether, and decided to opt for monthly subscription.

Estimated savings till March 2023: Nil

Tip: Are you paying month to month, or a slightly longer term? Does your host offer discounts if you pay for six months or annually? Many hosts, in fact, do!

The savings add up

Needless to say, the steps taken by Lata helped her save some money. In the first part of this series, we saw how Lata was able to save 87 US dollars. In this part, we saw that her proactive steps yielded savings of another 30 US Dollars.

All in all around 120 US dollars saved upto March 2023, or about 40 US dollars a month, is quite a bit of money saved! Maybe she should get a hot, tempting hosting plan during a future sale this year to reward herself for her efforts. Just kidding.

Wrapping it all up

Under normal circumstances, a cost increase due to inflation is expected and mostly a given. Inflation may vary from low single digits to high teens, depending on the country in which one lives. But present times are challenging and dynamic, where double digit inflation is prevalent in many countries at the same time, even those who have not seen such costs in 20 or even 30 years!

Add to this the falling value of the British Pound, or the Euro, in comparison to the US Dollar. I already mentioned INR's fall compared to the US Dollar earlier. The story may not be too different where you live.

But these times also present an opportunity to reflect on the hosting situation and think of ways to keep a tab on the hosting costs. I hope Lata's approach gave you an idea or two. However, the story does not end there.

Some questions that may arise such as:

• Has this post covered most of the practical approaches? (i.e. ones that can be considered as low hanging fruits)?

• Did Lata miss something obvious?

• Can she optimize the costs further?

I leave you with these questions as food for thought, and may the Low End Spirit in you figure out the best way(s) for you.

Author's Note: Thanks to the two reviewers, who shall remain unnamed, for their feedback and suggestions for this post.

-By A. Vyas, January 2023

(This is a two part series that is aimed to help customers looking to optimize their web hosting costs, although many of the suggestions may apply to the web hosts themselves)

Introduction

It is early January 2023 as I write this post. The 2022 end of the year Holiday season has just concluded, the Black Friday/ Cyber Monday and Christmas/ New Year Sales have concluded, and hopefully many of you still have some money left over to pay for your renewing your web hosting plans! This period becomes allows some time to reflect on how much one is spending on subscriptions in general and web hosting services in particular. The possibility of an economic downturn is one reason. The other factor, which I have mentioned below, is already a reality.

Over the past few months, many web hosting providers have increased their hosting fees on account of rising costs. European countries in particular have witnessed very high energy costs, which has added to the problem of high inflation. This has resulted in higher costs for running their business. In most cases, they have no choice other than passing on these increases to their customers.

In such a situation, how can a user, who is say a typical customer of Low End hosting, keep the hosting costs low? Let us take a look at five different ways to help you keep a tight leash over your hosting costs in the coming months.

Web Hosting Plans- The Problem of Plenty

Before we get down to business, let us try and answer this all important question:

Do you have too many hosting plans, or do you have a large collection of hosting subscriptions?

If the answer is yes, you are in a perfect position to take a deep, hard look at your respective hosting collections. There may be some impulse purchases, and some plans that do not make any sense now. Those are ripe to be dropped or culled. For example,

Is there a 2 GB shared hosting plan that you purchased during last Black Friday? Or an older OVZ server that is still active?

If the answer is no, you are in a good position to start optimizing further- maybe switch a plan or two to a lower tier. We will discuss that option in Part II of this series.

Where and How to begin reducing your web hosting costs?

Lata's Approach to Saving on Web Hosting

Let us take a look at the Web Hosting "Empire" of a person called Lata. She has gathered quite a number of webhosting plans over the years. When she was thinking of reducing these expenses she adopted the following approach.

First, she prepared a chart that lists all the hosting plans she actively subscribed to. In Lata's case, as might be the situation with many of you, this presented quite a collection of Shared hosting, Reseller, NAT, OVZ, KVM and Storage servers. In addition, she also has subscriptions for many SaaS (Software as a Service) based hosting plans from providers like Jimdo and Shopify.

In other words, short of a Dedicated server, Lata had subscribed to practically every sort of hosting plan one may typically find here on Lowend Spirit(https://lowendspirit.com/). While some of the hosting plans have been procured for friends and family, others are for her customers. Finally, a few plans are for her own personal or hobby projects.

_{Lata's Webhosting collection}

Let us see how Lata began to arrive at optimizing her costs.You can also develop a method that suits your situation the best. Lata tried five different approaches to arrive at some cost savings. In this first part of the series, we will look at three approaches. The balance will be covered in the second installment.

For sake of simplicity, all costs are mentioned in US Dollar terms, and any savings are projected to occur between January and March 2023.

I. Consolidate the hosting plans

Once Lata drew up her "hosting empire", she added up the fees for these plans. She sorted them by fees incurred, starting from the highest to lowest. Most of her plans were paid annually or for a longer duration. Many were up for renewal over the next couple of months. This helped her to start prioritizing which hosting plans she should close. Terminating the most expensive plans alone could have added to her savings quite a bit, had it been a practical approach.

Estimated savings till March 2023 : about 40 US Dollars

Tip: List down the pricing for all the plans you have, that alone will be a great starting point before you start looking at savings.

II. Moving the hosting to a cheaper location(s)

Hosting in Europe typically used to be cheaper till recent months. Generally speaking, this is likely to remain for some time, depending on how the current energy crisis evolves. However, these times also present an opportunity to move the hosting to a different location - say North America. And, in some (very rare cases) also Asia.

Lata followed this approach and reached out to some of her hosting providers who had services across multiple regions. She asked them,

"Would they consider moving some of her services to cheaper locations?"

Some providers agreed, others refused, a couple of them offered to move her for a fee. Lata did a quick cost to benefit analysis, and decided that some non-critical service(s) could be moved to cheaper locations. However, the mission critical stuff that required the lowest latency or higher reliability remained at their present locations.

Estimated savings till March 2023 : about 35 US Dollars

Tip: Do you have hosting plans with provides who have services across multiple regions? Maybe a chat with the support team could help you figure out some savings.

III. Lower the specifications a notch or two

The third approach Lata adopted involved asking an all-important question:

Did she really needed the plans with higher specifications- hosting she had subscribed to "Just in case"? 

For example, she had a shared hosting plan in North America with about 20 GB of Storage space, with a maximum of 10 domains, and 1 TB bandwidth per month. Most of this capacity was lying unused. She decided to move to a lower plan that had half the specifications. This resulted in nearly 30 percent savings.

Estimated savings till March 2023: about 12 US Dollars

Tip: In times of tightening the belt, scaling down a plan or two can help save some costs.

The savings do add up, and the journey is not yet complete.

By adopting the above three approaches, (consolidating the hosting plans, moving the hosting plans to cheaper locations, and lowering the specifications on the plans) Lata is projected to save around 87 US Dollars over three months (or about 30 US dollars a month) in hosting fees. And she was not done yet! In the second part of this series, let us explore some more ways in which you can learn form Lata's experience and determine ways to lower your own web hosting costs.

Note: This article was originally written in October 2022, the estimated savings and timelines have been revised to reflect the costs at the time of publication.

Welcome to this first post of 2022 on the LES Blog!

Nearly 14 months have passed since the seminal announcement by Red Hat about the End of Life (EOL) for CentOs8. The much loved server oriented Linux distribution did reach it EOL on December 31, 2021. Over this period, how have the users coped up? What are the alternatives they have settled for? Did the predictions of doom and gloom really come true? Finally, which of the several alternative projects have made the greatest headway or found the highest user acceptance? Read on to learn more.



Source: Wikimedia Commons

You can listen to the audio version of this post at (External Link)

In This Post

1. Background: RedHat Discontinues development for CentOs8
2. Every Adversity is an Opportunity
3. Do We Have "Good" Alternatives?
4. Alma Linux and Rocky Linux
5. Other Alternatives to CentOs8
6. What About CentOs Stream and CentOs7?
7. Wrapping Up: End of CentOs8 Era
8. Optional reading - For Fun's Sake

Background: December 2020- CentOs project closing its doors

In early December 2020, the folks at Red Hat Linux had made the announcement that CentOS 8, the latest version of the open source server oriented operating system, would reach its end of life on December 31, 2021. Development of Centos 9 would stop. CentOs project would focus on something called CentOs Stream, instead of a server distribution that was downstream of Read Hat Linux.

Naturally, there was a lot of hue and cry among the users of CentOS in particular, and the open source community in general. The recurring view expressed was that an open source project that was backed by a large company, which decided to shut it down, leaving the users abandoned. The large company, Red Hat Linux, had itself been acquired by a much bigger tech company, - none other than Big Blue a.k.a IBM.

Every Adversity is an opportunity?

Since the day of the announcement, several expected and un-expected things happened.

a. There was an uproar in the tech community, and discussions on Reddit and web hosting forums, including this one.

b. Two new major projects were announced: Rocky Linux and Alma Linux. Both are backed by well known names in the Linux world. The trend of announcing new alternatives to CentOS continues. As recently as January 2022, SuSe Linux have proposed their own alternative to CentOs, called Liberty Linux. And if you scroll below to the "Piece De Resistance" section below, you may be in for a surprise.

c. Media houses, publications, bloggers and YouTubers all began to come up with "Alternatives to CentOs" type of posts. Some posted good alternatives, others simply drew up a random lists of operating systems which included Windows 10!

d. Competitors such as Oracle began to run promotions trying to woo away users into the fold of Oracle Linux.

They say, every adversity is an opportunity. Maybe for the real change makers over the long run. In reality, in the short term, it is the opportunists who can also benefit.

What turns out to be true in this case? Read on to find out...

It's good to have alternatives... but do we have "Good" alternatives?

My main intention here is to see where things stand nearly 14 months after the Red Hat announcement was made. In early February 2022, we will be in the fifth or sixth week since CentOs 8 reached its end of life. given this timeline, some of the questions I was keen to understand were:

a. Many alternatives to CentOs were proposed as in the multiple lists prepared by different publications. You can read some of them here and here. Have the users really migrated to one of the many alternatives that are that were proposed?

b. What made them choose one option over another? Did any move to CentOs Stream?

c. What is the feedback about these alternatives? Which one really gained traction among the users?

Taking a Look: User feedback and beyond

Alma Linux and Rocky Linux - The Dynamic Duo

While researching for this article, it has become evident that the two most popular CentOS alternatives are AlmaLinux and Rocky Linux. The former is created by the folks at Cloud Linux, while the latter is set up by one of the original co-founders of CentOS.

No matter which publication or resource once one looks at, these two names keep coming up right at the top.

Source: Alma Linux

Alma Linux has a very good comparison of features, organizational structure, etc. for some of the alternative If you scroll below, you will find a table comparing the different alternative: CentOs, Alma Linux, Rocky Linux, Oracle Linux and Red Hat . Alma Linux is an open a community based distribution from the folks at Cloud Linux. Rocky Linux is a private limited company started by one of the original co founders of CentOS. The former is a nonprofit, latter is a for profit setup. They both offer scripts that help the users migrate from CentOS to their respective distributions. Compatibility with Red Hat Enterprise Linux support, end of life cycles etc are pretty much identical.

Source: Alma Linux

Some of the limitations of the alternatives were discussed extensively across Reddit, HackerNews and other sources. Some of the below also came up in this discussion right here on LES. They include, in no particular order,

• Community, or lack of
• Longevity of the teams or support from the organizations behind the projects
• Maturity of the products
• Backing by large Corporates. Some people were not happy with Corporate support for some of the newer projects- the Red Hat experience probably left a bad taste.

Computingforgeeks has a good list of features that discusses the pro's and con's of the different alternatives to CentOs. You can also read this discussion on Reddit and on Hackernews. For those who prefer the Video version, here is a decent resource that explains the difference between Rocky and Alma Linux.

Update April 2022: Distrowatch.com in their April 4, 2022 edition of weekly newsletter (Issue 962) mentions an article by FOSS Force which talks about Alma Linux as the most favored replacement for CentOs.

The Next most discussed/ talked about alternatives to CentOs

Based on anecdotal experience and discussion with the LES community, I did not encounter many users who have taken up any of the other alternatives. This includes Oracle Linux. Granted this is a small subset of the user base, it does give a data point to be considered. With that in mind, I am listing a few alternatives that I have come across nonetheless.

Oracle Linux

Oracle Linux is based on Red Hat Enterprise Linux. Oracle even created a page on its website targeting CentOS users. They have developed a script that can convert existing CentOS installations to Oracle Linux. This distribution is free to use, but support contract is available for a charge.


Source:Wikimedia commons

According to the Wikipedia page for this distribution,

Oracle Linux is ..."available partially under the GNU General Public License since late 2006.[4] It is compiled from Red Hat Enterprise Linux (RHEL) source code, replacing Red Hat branding with Oracle's."

Liberty Linux (project by Suse Linux)

This is the newest kid on the block, I had posted about it in the LES forum. Too early to discuss or comment, pending further updates, I will leave you with the link to the LES discussion.

Springdale Linux

This is a project by the members of the computing staff of Princeton University and the Institute for Advanced Study, this was earlier known as PUIAS Linux. They describe themselves as a "Custom Red Hat based Distribution and Mirror." Among other things they have an Interesting Webpage !! The design reminds us of the early years of the World Wide Web.

visit Springdale Linux to learn more.

ClearOS

ClearOs offer commercial and community edition, as per the discussion on ClearOS forums, this may not be a long term viable alternative

VZLinux

This is a CentOs based distribution from folks at Virtuozzo. You can download the image files from:
https://repo.virtuozzo.com/vzlinux/8/iso/

Other than the above, I did not find much written or talked about this alternative to CentOs.

Fedora Server

Fedora is primarily a desktop oriented distribution, and a server version of Fedora, while possible and even capable, does not make a lot of sense. Atleast to me. According to one of the posts I came across on the topic,

Fedora server also follows short lifecycle. It is a community-supported operating system. Paid or commercial support option does not exist, and the lifecycle makes it less appealing as an alternative.

What about CentOs Stream?

Would you really consider moving to CentOS Stream for your servers? I have not found a definitive "Heck Yes" or "Hell, No!" answer to this query. Nor have, it appears these guys.



Source: Postgresweb

Are you perfectly happy with CentOs 7?

If you are happy with CentOs7, or belong to the if-it-ain't-broke-don't-fix-it school, then you have atleast 30 plus months of updates for CentOs coming from Red Hat. The Frequently Asked Question page on CentOs site confirms the same. In that case, should the EOL for CentOs 8creally be a concern for you?

And now, the pièce de résistance: A server Linux distribution from Microsoft

This one is real guys, and no more a joke in the Linux and open source communities. (Anybody remember Lindows.. er.. Linspire?)
edit: The project is still alive! That's cool.

Source: Wikipedia

The Linux distribution from Microsoft, called CBL-Mariner, is based off Fedora and Linux From Scratch. Form the github page for CBL-Mariner, https://github.com/microsoft/CBL-Mariner

CBL-Mariner is an internal Linux distribution for Microsoft’s cloud infrastructure and edge products and services. CBL-Mariner is designed to provide a consistent platform for these devices and services and will enhance Microsoft’s ability to stay current on Linux updates. This initiative is part of Microsoft’s increasing investment in a wide range of Linux technologies, such as SONiC, Azure Sphere OS and Windows Subsystem for Linux (WSL). CBL-Mariner is being shared publicly as part of Microsoft’s commitment to Open Source and to contribute back to the Linux community. CBL-Mariner does not change our approach or commitment to any existing third-party Linux distribution offerings.

Wrapping it up: Nearing the end of the CentOS Era

It is great to see several alternatives for CentOs, which is one of the beauties of the Open Source world. The challenge, as always, is that it presents a problem of plenty. Many folks in the LES Community- web hosts or the advanced users who used CentOs previously, seem to have converged towards Alma Linux or Rocky Linux. Among the rest of the list- Oracle Linux might have some users, given that many members on both the green forums have atleast one instance of Oracle Free tier.

I hope that this post has captured the "After the Storm" scenario since CentOS8 reached EOL. Over the long run, I intend it will offer a handy resource for the LES community. In addition to the top two alternatives (Rocky Linux and Alma Linux), there is a long tail of "also consider" alternatives. Which is one of the real beautiful things about the Linux and Open Source world.

Further (Optional) reading: FFs (For Fun's Sake)

While researching for this post, the search results threw some junk posts. I am adding a few of them below to add a flavor of humor. For example, really useless practices by some of the reviewers and authors. These included:

Some reviewers did a desktop review of Alma Linux, as in literally installing graphical interfaces. They spent a lot of time talking about how GNOME performed on this distribution versus KDE. One person even, did a screen capture of games being played on Alma Linux. Not sure if any server admins run GUI or play Steam games on their production servers.

Source: HowtoShout

The Techmint guys aren't too far behind : https://www.tecmint.com/wp-content/uploads/2021/06/AlmaLinux-Desktop.png

Windows 10 as an alternative to CentOS? Really??
g2 smokes some cheap stuff.

The Roll your eyeballs version
Simply copy and paste names of popular Linux distributions, and not really doing any favor to the reader. Hostingseekers, you need therapy

Author's Note:

I put together this post with the intention of creating a handy resource for readers here on the Low End Spirit blog. I am not a user of CentOS or any of its derivatives. Even though I began my Linux Journey with Red Hat 6 (Zoot), my poison of choice for many years has been Debian. Or, grudgingly, I use Ubuntu LTS where the Software as a Service (SaaS) vendors require me to use it.

This post presented an opportunity first and foremost to understand what is going on in the CentOS world, or the CentOS extended universe. (To borrow a phrase from the Marvel Universe). I have not done any installation or testing of my own for the purposes of this article.My interest was mainly in understanding what has been the market acceptance over the past few months. One hopes that has been served well.

For any questions or comments, write to me at contact+les@amarvyas.in

1- Have your vm already installed with one of the provided templates e.g: debian 9. ssh into your vm as root install the ipxe package:

# apt-get install ipxe
# update-grub

Get your ip, gateway from your vm info email, provider cp or on debian this way and store it as it will be needed later.

# cat /etc/network/interfaces
or
# ip r

# reboot

2- From your vnc or console, you should be able to see similar grub menu

after you press enter then quickly keep pressing CTRL+B ,or you can set to boot from CD only in you cp... then you should

enter your ip info you stored from the above step 1 like show in next image

if your vm supports ipv6 then it may try to configure it so you press enter like shown:

let it boot and should be good

I wrote this article when i got a HH vm and uploading iso is broken at time of this article.

References :
1- https://unix.stackexchange.com/questions/190865/is-it-possible-to-add-some-pxe-network-boot-option-to-grub
2- http://www.panticz.de/Install-GRUB-iPXE-netboot

There are couple of providers offerings NAT vps such as inceptionhosting , mrvm and webhorizon .

Thanks to webhorizon for sponsoring this guide and offered to bump my ram.

NAT VPSs are affordable and useful for practical small apps. In this guide we will setup a docker swarm cluster with a mix of single KVM manager and couple of OpenVZ7 NATs workers. For high availability it is recommend to have an odd number < 5 managers.

Minimum requirements:
- 1 KVM and 1 NAT nodes with at least 512MB Ram and a minimum of 5GB disk
- using IPv4 on all nodes.
- The host OS is Alpine Linux version 3.15 which is latest at time of writing this guide. Webhorizon already provides alpine as an os template on their NAT range.

Section 1: ( Host setup and applications install )

Host kernel settings:
- All hosts should have those kernel settings on ; For NATs if not enabled then open a ticket and nicely ask.

# cat /proc/sys/net/ipv4/ip_forward
1
# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

for KVM you can add that by

cat > /etc/sysctl.d/01-configs.conf <<EOF
# alpine
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
EOF

If your NAT provider does not include Alpine Linux template then one can install it using this script and my slightly modified version script.

On your NAT vps cp make sure you enabled

OpenVZ7 Install wireguard-go

# apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing wireguard-go
# apk add wireguard-tools

while the above is a testing repo so far it has been stable. You can always compile your own version.

KVM Install wireguard

# apk add wireguard-tools

Install and enable Docker

# apk add docker
# service enable docker

Section 2: Nodes details

1. Server "kvm"

   • Main Interface IP = 185.253.47.54
   • Wireguard Private IP = 192.168.1.2

2. Client "OpenVZ NAT"

   • Main Interface IP = 10.37.136.225
   • Wireguard Private IP = 192.168.1.10

Section 3: ( Generate keys and Conf file )

Wireguard requires key generation and setting up the network interface in a certain way. There are scripts and quick command lines to accomplish this task. The next details describe the manual method. Take a note of the cat outputs tp add in wg0.conf file.

generate keys those way on both nodes

# wg genkey | tee privatekey | wg pubkey > publickey
# cat publickey
     OaJ6KDPdgOs+aOJh7oEnaQYs/w9aIuHd0LfFgyx3ORw=
# cat privatekey
     WH8fN46zIMIXCxt2m5+I75vI+31oehh6aHte3qYYIys=
# # presharedkey is only needed for client node
#  wg genpsk > presharedkey
     Gg9HaCMkeEzUNl5zz9fUEzaDKIO2MNUKeG3D+ihchS8=

Server side config

# mkdir -p /etc/wireguard
# nano /etc/wireguard/wg0.conf
     [Interface]
     ListenPort = 51820
     PrivateKey = WH8fN46zIMIXCxt2m5+I75vI+31oehh6aHte3qYYIys=
     
     [Peer]
     PublicKey = OaJ6KDPdgOs+aOJh7oEnaQYs/w9aIuHd0LfFgyx3ORw=
     PresharedKey = Gg9HaCMkeEzUNl5zz9fUEzaDKIO2MNUKeG3D+ihchS8=
     AllowedIPs = 192.168.1.10/32

Client side config ; change port number to one in the range given to you, in my example ist 22520

# nano /etc/wireguard/wg0.conf
     [Interface]
     ListenPort = 22520
     PrivateKey = +KoN0pd30I6WERz2s9itHlIoikZBh1q2cbhB88UsnlI=
     
    [Peer]
    PublicKey = OaJ6KDPdgOs+aOJh7oEnaQYs/w9aIuHd0LfFgyx3ORw=
    PresharedKey = Gg9HaCMkeEzUNl5zz9fUEzaDKIO2MNUKeG3D+ihchS8=
    AllowedIPs = 0.0.0.0/0
    Endpoint = 185.253.47.54:51820
    PersistentKeepalive = 25

Section 4: ( Update network configuration )

Server side interface file

# nano /etc/network/interfaces
    ...
    auto wg0
    iface wg0 inet static
            address 192.168.1.2/24
            netmask 255.255.255.0
            pre-up ip link add dev wg0 type wireguard
            pre-up wg setconf wg0 /etc/wireguard/wg0.conf
            post-up ip route add 192.168.1.0/24 dev wg0; /sbin/iptables -A FORWARD -i wg0 -j ACCEPT;/sbin/iptables -A FORWARD -o wg0 -j ACCEPT;/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
            post-down ip link delete dev wg0
            post-down  /sbin/iptables -D FORWARD -i wg0 -j ACCEPT;/sbin/iptables -D FORWARD -o wg0 -j ACCEPT;/sbin/iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

Client side interface file

# nano /etc/network/interfaces
    ...
    auto wg0
    iface wg0 inet static
        address 192.168.1.10/24
        pre-up /usr/bin/wireguard-go wg0
        post-up /usr/bin/wg setconf wg0 /etc/wireguard/wg0.conf
        post-up /sbin/ip route add 192.168.1.0/24 dev wg0;
        post-down /sbin/ip link del wg0
        post-down rm -f /var/run/wireguard/wg0.sock

Section 5: ( Reboot and Test )

Do a reboot and start testing the connection for both server and client, yours will be different when changing the ips, ports and keys etc...

# ip a

# wg show

above test show connection is made and our wireguard nodes are reachable.

This is the first in a series of posts in the "Easy Mode" series

I thought I would write this up as a super easy basic guide to an almost no-effort method of migrating a KVM VPS from one host to another without having to worry about any minor changes in software stacks that could cause your backup and restore to fail or not restore as expected.

As a host one of the unfortunate realities is that sometimes it is just necessary to have migrations either due to end of life of an OS major version or impending hardware failure etc and on the rare occasion you need people to migrate their own server, this usually results in a 10% panic rate and a desperate cry for help.

With all that in mind I thought I would illustrate possibly one of the most simple server migration methods possible, I have done this based on KVM to KVM as that is likely to be the most common method however this method should also work for Xen HVM to Xen HVM or KVM to Xen HVM or Xen HVM to KVM and finally VMware to VMware should also work fine.

I can't cover the huge amount of different control panels and access methods out there but all that I have used either have access to an ISO mount or a rescue mode so it should be easy for you to use this guide as long as your host's control panel provides at least an ISO mount option or a rescue mode.

In this example, I have used a KVM VPS on a host that uses SolusVM and migrated it to a host that uses Virtualizor.

Requirements and additional info before we get started.

• The destination VPS should have the same size hard disk (bigger is fine).
• This will only go as fast as the link between the 2 locations, latency is a huge factor, moving a 50GB Disk image from India to New York is no fun.
• Some attention to detail is required, it is possible to lose your data if you mess up the commands.
• If the destination control panel does not provide a 'reconfigure networking' button or a serial console/VNC option you can use to reconfigure networking yourself you may want to consider configuring but not activating the network settings for the destination VPS on the source prior to shutdown, SolusVM, however, will do this for you and in some cases depending on the Hosts configuration options in Virtualizor it will also.
• Source VPS, in this case, was at Inception Hosting - London, the destination VPS was with Nexus Bytes - New York

First of all, I created a quick KVM VPS as the source on the SolusVM based host, installed apache2 and slightly modified the test page which you can see loading on the source IP:

Then I shut down the Source VPS and put both the source and destination VPS into rescue mode.

For SolusVM (Source):

For Virtualizor (Destination):

note: I noticed virtualizer has a tendency to just lock out all options even those for coming out of rescue mode when you do this which is resolved by simply refreshing the page, don't worry it does not resubmit the action

Now you should have both the source and destination in rescue mode and you should be able to login to both over ssh:

SolusVM VPS (Source)

Virtualizor VPS (Destination)

Next, you need to examine your disks to make sure you get the right one, the rescue modes will create a ramdisk so what was vda or sda on your VPS when it was running may not be now, this is the part you need to be sure you get correct, I have shown a side by side below using fdisk -l to check which disk I want to migrate to which disk on the other end.

From the image above we can see that the 5GiB disk is the one I want /dev/vda because I know my source disk is 5GiB and on the destination side I can see that it is actually /dev/vdb I want to copy the disk to, the 10 GiB disk. It is very important that you check this before going any further as it could well be different.

Because we now know I want to copy vda on the source to vdb on the destination we can structure the command to do this.

dd if=/dev/vda bs=32M status=progress | ssh root@45.61.123.123 "dd of=/dev/vdb"

let us break this down so it is better understood as there is nothing worse than knowing which buttons to press to achieve a goal, but having no idea why they work, or what they actually do:

• dd - Short for "data duplicator" a common commend used for copying or duplicating data.
• if=/dev/vda - The input or what we are reading, in this case, the entire virtual hard disk image
• bs=32M - BYTES, the number of BYTES to read per block when copying, you can set this yourself, I have found 32M is a good average.
• status=progress - This shows the progress of the copy, it has a very small overhead but without it you get nothing in terms of a progress indicator
• | - The pipe, in very simple terms, lets you pass the result of one command or set of commands to the next
• ssh root@45.61.123.123 - Because this is after the pipe | it takes what was done before and sends it over an ssh session as root to the destination IP 45.61.123.123 (fake)
• "dd of=/dev/vdb" - Finally the instruction on what to do at the other side which is to use dd "data duplicator" to write the blocks it is receiving over ssh to /dev/vdb

Once that command is issued you will see something like this:

And when it completes you will see:

As we can see that took 634 seconds or a little over 10 minutes to complete and now the disk image on the destination should be identical to the source.

The next step is to cancel rescue mode on the destination via the control panel and then boot the destination VPS, going to the new IP address once it is finished booting shows that it was a success as the same page loads on the new IP:

If your destination VPS has a bigger disk size than the source you will not instantly gain the extra space by following the above process, you will need to either expand your partitions or extend your logical volume if using LVM, that however is beyond the scope of this particular tutorial but if you are also looking for easy mode for that then just boot the VPS with a GParted ISO or a rescue cd that contains GParted which will give you a simple UI to expand your partitions or look up vgextend and lvextend if using LVM.

If having read this you discover that your host does not have a built-in rescue mode then I would suggest requesting they supply a SystemRescueCd ISO (Also contains GParted) https://www.system-rescue-cd.org/ so you can mount it and boot into rescue mode.

I. Before We Start

We need to obtain our basic network configuration from our provider. Or, if we are running our own host node, we need to assign basic network configuration to ourselves. Our basic network configuration might look something like this:

For IPv6, one might expect something like:

But occasionally, IPv6 could be something like:

Notes:

• The /28 in the IPv4 address and the longer netmask are different ways of providing the same information about the size of the local, directly connected network. It suffices for us to have this information in one format or the other. We don't need both formats because the information is the same. Also, the broadcast IP might not be provided, since it isn't strictly necessary.
• For the second format of the IPv6 address, what happened to the /64? 😱 The /128 in the second form of the IPv6 address might seem clueless to IPv6 fans expecting a /64. Also, the second format of the IPv6 address includes a gateway6 address. The gateway6 address might seem strange to some IPv6 fans, but we need the gateway6 for our minimal, static configuration. More on all this below.

II. Introduction

In the previous post of this series we finished using the Proxmox web GUI to install our new Debian KVM VPS via the Debian netinst installer iso image. The final step in Part II was removing the netinst install iso image from the emulated cdrom and then reooting our new VM, which came up from its own internal filesystem:

In today's post, we continue from this exact place where we left Part II -- connected to our newly installed and newly rebooted KVM via the Proxmox web GUI. In this post, we will accomplish the networking configuration which was skipped in Part II because the Debian netinst iso doesn't automatically configure out of band IP addresses.

There are three network configuration and related tasks we will accomplish today:

• First, we go "inside" our VM through the Proxmox web GUI's emulated "physical" console connection and set up networking. In Debian, networking setup requires that we adjust the file /etc/network/interfaces to tell our VM its network address and the address of its gateway to the internet.
• Second, we edit the file /etc/resolv.conf to tell our VM the numerical addresses of Domain Name System ("DNS") servers it can use to translate human readable Uniform Resource Identifiers (URI) into numerical Internet Protocaol ("IP") addresses.
• Third, we set up /etc/apt/sources.list to tell our system's Aptitude software package manager ("APT") where to get software updates and the additional software packages we will want to install.

Section III, Quick Setup, runs quickly through all three of today's tasks in "recipe style."

Section IV offers additional context on our setup environment.

Sections V, VI, and VII provide additional details on today's three setup tasks.

Section VIII discusses security.

Section IX discusses backup.

When we finish the Quick Setup, our new Debian KVM VPS should be connected to the internet, DNS should work, and we should be able to use the Debian package system to add whatever additional software we want.

When we finish all of today's post, we should have reasonable context within which to understand our Debian VM's networking setup.

III. Quick Setup

Logged into our VM through the Proxmox web GUI, we run the command ip link show. This command will give us the name of our network interface, probably something like "ens18."

As root or with sudo, we edit the text of the file /etc/network/interfaces so that it contains the minimum necessary information:

auto ens18
iface ens18 inet static
  address IPv4_ADDRESS/CIDR
  gateway GATEWAY_ADDRESS

iface ens18 inet6 static
  address IPv6_ADDRESS/CIDR
  gateway GATEWAY6_ADDRESS

Using our example network configuration, our minimal /etc/network/interfaces looks like this:

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
  gateway 172.16.164.1

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
  gateway fe80:xxxx:xxxx:xxxx::3

Second, we edit the /etc/resolv.conf file so that it looks like this:

nameserver 1.1.1.1
nameserver 8.8.8.8
nameserver 2606:4700:4700::1111
nameserver 2001:4860:4860::8888

Third, we edit /etc/apt/sources.list so that it looks like this:

deb http://deb.debian.org/debian buster main contrib non-free

deb http://deb.debian.org/debian-security/ buster/updates main contrib non-free

deb http://deb.debian.org/debian buster-updates main contrib non-free

Finally, we restart networking so that our new configuration takes effect:

systemctl restart networking

At this point, we should have both IPv4 and IPv6 connectivity, and DNS and APT both should work.

IV. More Context

• Virtualized Console Connection

The Proxmox web GUI virtualizes a wired console connection. In other words, our web browser does connect over the internet to our Proxmox server, but, the view from inside our new KVM is the same as though a wired connection was attached. Our new KVM thinks it's talking over a wired connection to a physical console. From inside our new KVM, there is, as yet, no network connection.

By default, the Proxmox web GUI works via VNC. In the Proxmox wiki on serial terminal Proxmox warns that VNC might

not have the features you need (i.e. easy copy/paste between other terminals)

or it might be

impossible to capture all [kernel messages, standard output, or error] messages on [the] VNC screen.

Yep, copy / paste commands do not seem to work in the Proxmox KVM virtual console.

Also, if you enjoy using the vi editor, you might find what looks like a "Send-Esc" button among the set of choices within the set exposed by the top button on the console VNC control bar. Use of the real keyboard Escape key results in exiting full screen. However, a second real Esc seems to produce the expected mode change, despite that maybe we no longer can see too well without returning to full screen.

• No DHCP, No SLAAC

These days most network setups use Dynamic Host Configuration Protocol (DHCP) to autoconfigure IPv4 networking. The machine on which networking is to be configured asks for and receives from a DHCP server all the needed information for the networking setup.

It is possible to configure DHCP so that it always returns the same IP address to each VM, but, since our entire Proxmox network is static, it may be simpler to set up networking manually--the traditional way for servers.

Stateless Address Autoconfiguration ("SLAAC") provides automatic configuration of IPv6 addresses. SLAAC requires a /64, which is why people say, for IPv6, that a /64 is expected and that less than a /64 is clueless. However, it remains possible to hand configure a single static IPv6 address, as we are doing here.

What if, for whatever reason, we simply do not want to use SLAAC? What if our provider doesn't receive enough IPv6 addresses from his provider to allow passing on to each VPS its own /64? What if our provider's provider charges an extra fee for extra IPv6 addresses, but we do not want to pay our provider's pass through of his provider's extra fee? What if we simply choose to use single, static IPs as is traditional for servers?

• No Cloud-Init

As mentioned in the previous post of this series, most VM network setups these days are done with Cloud-Init. Proxmox supports Cloud-Init, which enables both networking and ssh access to virtual machines to be set up on the Proxmox hypervisor and outside of the VM. Cloud-init can use DHCP. Here, however, we have chosen the simplest possible manual configuration with static IPs.

• Our Static, Routed Configuration And Out of Band Gateway From Our Provider's Provider

Here, our single, static IPv4 and single, static IPv6 are each derived from a routed subnet assigned to our server node. However, our internet gateway IPv4 address is not included among our server's routed group of IPv4s. This is called an "out of band" gateway.

Besides routed subnets, it also is possible for a datacenter to assign to servers non-routed, individual IP addresses. Data for these non-routed IPs moves between the datacenter switch and server nodes via the "link layer." Hetzner has a tutorial on Debian network configuration which includes discussion of "bridged configuration" for non-routed IPs.

• Systemd in Debian Networking

Since about 2014, networking is setup on Debian with systemd. The choice of systemd initially was and has continued to be divisive. Nevertheless systemd has remained as the Debian default.

There are at least two basic variations of Debian's systemd network arrangement. The first--which seems to be the default variation for Debian systemd network configuration--at least with the netinst iso--is using systemd's networking.service. For example, by using systemctl, we can confirm that networking.service is what is being used on our Node:

root@Proxmox-VE ~ # systemctl status networking.service
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: 
   Active: active (exited) since Wed 2021-06-02 19:13:13 UTC; 1 weeks 2 days ago
     Docs: man:interfaces(5)
 Main PID: 791 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/networking.service

 [ . . . ]
root@Proxmox-VE ~ # 

Our test KVM also seems to think its networking is controlled by systemd:

root@debian-kvm:~# systemctl status networking
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2021-06-16 01:20:45 UTC; 4min 51s ago
     Docs: man:interfaces(5)
  Process: 448 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
 Main PID: 448 (code=exited, status=0/SUCCESS)

Jun 16 01:20:45 debian-kvm systemd[1]: Starting Raise network interfaces...
Jun 16 01:20:45 debian-kvm systemd[1]: Started Raise network interfaces.
root@debian-kvm:~#

As we can see, systemd networking.service calls the traditional debian ifup and ifdown.

root@debian-kvm:~# cat /lib/systemd/system/networking.service
[Unit]
Description=Raise network interfaces
Documentation=man:interfaces(5)
DefaultDependencies=no
Requires=ifupdown-pre.service
Wants=network.target
After=local-fs.target network-pre.target apparmor.service systemd-sysctl.service systemd-modules-load.service ifupdown-pre.service
Before=network.target shutdown.target network-online.target
Conflicts=shutdown.target

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

[Service]
Type=oneshot
EnvironmentFile=-/etc/default/networking
ExecStart=/sbin/ifup -a --read-environment
ExecStop=/sbin/ifdown -a --read-environment --exclude=lo
RemainAfterExit=true
TimeoutStartSec=5min
root@debian-kvm:~# 

The second Debian systemd possibility--not the default on Debian netinst.iso and not used here--is systemd-networkd. Sahitya Maruvada has a simple, clear, Debian systemd-networkd introduction, Working with systemd-networkd. The systemd-networkd wiki page and the systemd.network manpage also are available.

• Official Debian Network Setup Instructions

Official Debian network setup instructions include the Wiki, the Handbook, manual pages such as man interfaces, /etc/network/interfaces examples online, and sometimes locally:

# less /usr/share/doc/ifupdown/examples/network-interfaces

• The ip Command Usually Is Available Even Though Networking Setup Varies Among Linux Distributions

Setting up networking, DNS name resolution, and software package management is very different in different Linux distributions. Therefore, we should not assume that the steps taken below would be exactly the same with a different Linux distribution than Debian.

Nevertheless, despite the different distributions' differing network setup systems, the ip command, supplied by the iproute2 collection, usually is available these days. Please see also Red Hat's IP Command Cheat Sheet

Because the ip command often is available, networking can be configured in many distributions, including Debian, by running a sequence of ip commands. The net effect of the sequence of ip commands can be to get the network functioning on most distributions without touching that individual distribution's network setup scheme.

Here's an example of the ip command used in the context of an iPXE boot. Note that the first command in the linked example requires knowledge of the name of the interface. We can list the names of the interfaces on our system by running the ip link show command.

One issue with using a sequence of ip commands is that the network setup fails to persist across reboots. However, we can place the ip command sequence inside a script which will be run automagically every time the server reboots. The sequence of ip commands in a script reminds us of the days before systemd, when scripts controlled all parts of the boot process including network setup.

Our KVM VPS's internal network configuration that we will be using below is similar to how LXC containers are configured in Proxmox. As will be seen below, Proxmox's LXC containers' network configuration adopts a variant of the "scripted ip command" approach, which also works inside Proxmox's KVM VPSes.

V. Our VM's Network Setup

• Interfaces

Our original /etc/network/interfaces file, the one installed by the netinst.iso, might look like this:

debian@debian-kvm:~$ cd /etc/network
debian@debian-kvm:/etc/network$ cat interfaces.original
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback
debian@debian-kvm:/etc/network$ 

Note that, in the default from the netinst.iso, /etc/network/interfaces.d is empty, so sourcing its files does nothing to the configuration.

debian@debian-kvm:/etc/network$ ls interfaces.d
debian@debian-kvm:/etc/network$ 

Now, let's edit /etc/network/interfaces to match our example network information from the above Before We Start section.

debian@debian-kvm:/etc/network$ cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
  gateway 172.16.164.1

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
  gateway fe80:xxxx:xxxx:xxxx::3

debian@debian-kvm:/etc/network$ 

The minimum required information does not include comments (lines beginning with #). Maybe we can make the rash and short-sighted assumption that we are not going to install anything which will want a file included from interfaces.d. The loopback interface might no longer be required (please see lines 17 and 18 in this file from Debian sources). Thus, for our example setup, the minimum /etc/network/interfaces might be:

debian@debian-kvm:/etc/network$ cat interfaces

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
  gateway 172.16.164.1

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
  gateway fe80:xxxx:xxxx:xxxx::3

debian@debian-kvm:/etc/network$ 

When configuring Debian LXC containers, Proxmox configures their /etc/network/interfaces files using added post-up and pre-down routes. Similarly, just for fun, instead of giving the gateway addresses in our /etc/network/interfaces,, we can manually add routes. Except for the initial post-up and pre-down these added lines mirror ip route commands that we could run manually to set up or take down networking without touching the /etc/network/interfaces file.

debian@debian-kvm:/etc/network$ cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
     post-up ip route add 172.16.164.1 dev ens18
     post-up ip route add default via 172.16.164.1 dev ens18
     pre-down ip route del default via 172.16.164.1 dev ens18
     pre-down ip route del 172.16.164.1 dev ens18

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
     post-up ip route add fe80:xxxx:xxxx:xxxx::3  dev ens18
     post-up ip route add default via fe80:xxxx:xxxx:xxxx::3  dev ens18
     pre-down ip route del default via fe80:xxxx:xxxx:xxxx::3  dev ens18
     pre-down ip route del fe80:xxxx:xxxx:xxxx::3  dev ens18

debian@debian-kvm:/etc/network$ 

VI. Our VM's DNS

We might want to add more or different nameservers to /etc/resolv.conf. Our Quick Setup configuration, above, includes IPs from Cloudflare and from Google.

VII. Our VM's Apt Setup

The Debian wiki instructions for configuring apt are at https://wiki.debian.org/SourcesList. There also is a man page. The configuration shown above, in Section III Quick Setup, is from the SourcesList Debian wiki page.

The Debian Security Information page says:

You can use apt to easily get the latest security updates. This requires a line such as
deb http://security.debian.org/debian-security buster/updates main contrib non-free

Many of the larger providers offer Debian mirrors. For example, Debian packages and security updates are available from the Hetzner Debian Mirror

After /etc/sources.list is edited, we update our system's package repositories as follows:

apt-get upgrade && apt-get dist-upgrade -y

We can see exactly which packages are installed by looking at the logs in /var/log/apt.

We may wish to install openssh-server so that we can connect to our VM via ssh in addition to our Proxmox VNC connection. With ssh we regain cut and paste functionality while enjoying lower apparent latency!

apt-get install openssh-server

The Kennedy article, mentioned below in Section VII, has some good tips for ssh server configuration.

VIII. Security

Google suggests its first choice among essential server security articles. This article from 2013, by Bryan Kennedy, seems to provide still-good advice, except that, nowadays, many people prefer to use ed25519 keys

IX. Backup

After all this work, we certainly want to make an offline backup of our new VM. We can use Proxmox to make the backup and then download a a copy from the host node's /var/lib/vz/dump directory.

This is a guest post by forum user @alexxgg,

Hi there!

These days when the internet shares things in milliseconds, who of you have needed to share a file quickly? I bet some of you remember at least one time one of those cases.

As you probably know, in terms of web servers, we can share files with popular software like Apache, Nginx, and Lighttpd but this software need basic configuration, also they consume server resources as long as they’re active.

What if you could set up a basic HTTP web server without installing Apache, Nginx, or Lighttpd? Well, that’s sounds kind of impossible, and even more unbelievable is that you can kill it with Ctrl^C. Thanks to Python3 we can do that with its HTTP server module.

This module will deploy an HTTP server in any directory of the server, even in the root directory and that sounds ridiculously dangerous, fortunately, the default port of this HTTP server isn’t 80. Instead, it will use port: 8000 but you can assign a custom port, very convenient for NAT environment instances.

Now you can deploy a basic HTTP server -in the current directory with the default port- by typing this single command line:

python3 -m http.server

Starting HTTP Server

Killing HTTP Server

HTTP Server as Shown in Browser

You can change the default port (example port: 32085) and specify a directory (example directory: /tmp/) with:

python3 -m http.sever 32085 --directory /tmp/

More information about Python3 httpserver module is available here

Also, there is a github page here

Note: of course, you will need to install Python3 in order to use these post example command lines.
Leave a comment to let me know any questions or suggestions.

This is a really QUICK TIP!

When you run any command as root, using sudo, the password is remembered for 15 minutes by default.
If you want to change the time that the password is cached, open the terminal (as root) and run:

editor /etc/sudoers 

Find this line in the file:

defaults env_reset

And change it into:

defaults env_reset , timestamp_timeout=x

where “x” is the time in minutes that the password will be cached.

Save and exit and work is done!

Cloudflare has recently adapted their SSL for SaaS product, which was originally limited only to their Enterprise customers but has now been released publicly. Yesterday they announced their Cloudflare for SaaS solution available to everyone. And to reflect their recent service evolution of the product they have re-named it to: Cloudflare for SaaS.

You may ask yourself: What exactly is SaaS?
Software as a Service (SaaS) is a method of providing software to an end-user where the software is not installed and maintained by the user, but via hosted services (most often through a web browser). Popular examples of such a service include Salesforce, Google Apps, Microsoft Office 365, etc.

Since the 15th of April, they have released their beta stage, which is available for sign up if you submit your request here. Before the introduction of Cloudflare for SaaS, your best bet was for your customer to set up a CNAME record and have them generate a private key and CSR.

On top of that, you would be required to maintain a solution to generate and securely store private keys. But with the introduction of Cloudflare for SaaS, it is now more freely accessible and has provided less of a burden when managing multiple customers.

Cloudflare Workers

Back in January when SSL for SaaS was announced, 80% of beta users were already building their application on Cloudflare workers. But by combining the use of Cloudflare for SaaS with Cloudflare workers, will overall reduce the time and core resources from building your application.

Security

It also has already provided DDoS Protection and Web Application Firewall (WAF) built-in. Alongside the benefits offered to the Enterprise customers at no additional cost.

So, if you want to have more control of your SSL certificate solution and would like a simple but manageable solution, then why not sign up for the beta.

When installing software on your VPS you will end up with both empty files and empty directories, often these are used as placeholders/lock files/socket files for communication.

This short guide will give you some examples on how to find those empty files/directories.

The command we are going to use is the “find” command. To find empty directories/files in the current directory, you use the parameter “-empty“.

You also have to use the parameter “-type” to define if you are looking for directories (d) or files (f).

Examples

Here is the command to find empty directories in the current directory:

find ./ -type d -empty

And here is the command to find empty files in the current directory:

find ./ -type f -empty

If you need to know how many empty files you have in the current directory, pipe the find command to “wc -l“:

find ./ -type f -empty | wc -l

Similarly, to recursivly count how many how many files are located under the current directory and sub-directories,  you can use the following command:

find ./ -type f -not -empty | wc -l 

To remove all empty directories in the current directory, the command you can use is:

find ./ -type d -empty -exec rmdir {} \;

– In all the commands above, the  (./) means the current directory or folder, if you want to perform actions in other directories, just replace the  (./) with the path to the new directory.

– In system directories such as /etc/, there are many empty files and directories.

But it is strongly recommended to not remove them.

What is Docker?

If you're reading this, most probably you already know Docker or have at least heard about it a lot. But still, for the uninitiated, Docker is an open platform for developing, shipping, and running applications and it enables you to separate your applications from your infrastructure so you can deliver software quickly. Find Docker interesting and want to know more? Head over to their docs and you can find all the information you need!

What is Portainer?

So, talking about the elephant in the room, Portainer is a fully-featured web based GUI management tool for Docker. It runs locally, giving developers a rich UI to build and publish container images, deploy and manage applications and leverage data persistence and horizontal scaling for their applications.
Worried about the cost? Portainer Community Edition is open source, free forever and used by more than 500,000 developers worldwide.

What can I use Portainer for?

1. Visualize your server's docker environment on your web browser. (I know that you don't fear the terminal, but hey, a little help won't harm anybody!)
2. Aggregate view of Docker Swarm clusters (Yeah, it's that fancy!)
3. Deploy containers with some pre-built templates, right from inside the Portainer.
4. Start, Stop, Kill, Restart, Pause, Resume and Remove the containers easily with the web-GUI.
5. Facing any issue while deploying containers? Don't worry, Portainer to rescue! You can inspect the logs for any containers directly from the GUI and see what is stopping you from conquering the world.

How do I install Portainer?

So, you're happy to give Portainer a go and want to know how can you install it? I have got you covered:

1. Make sure you have Docker Engine installed on your server. You can follow the install instructions given here.
2. Run the following command to create a docker volume which should give output as portainer_data indicating the command was successfully executed:

sudo docker volume create portainer_data

3. Once the volume is created, run the following command to create and run the Portainer container:

sudo docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce

The above command should create and run the Portainer container on your server's port number 9000, which can be generally accessed in following ways:

a) By opening http://your_server_ip:9000 in your favorite browser.
b) If you have a domain name pointed towards your server's IP, by opening yourdomain.com:9000 in your browser.

Note: If you want to access Portainer over a subdomain instead of every time typing yourdomain.com:9000, you can put it behind a reverse proxy with the help of any web server, like Caddy.

4. Alright, once the container is up and running, access it via any of the above methods and you will be greeted by the following initial setup screen of Portainer:

Set the username and password for admin user here and click on Create User.

5. Next, select Docker as the container management environment you want Portainer to connect to (yes Portainer can connect to Kubernetes too, but that's a story for another day):

6. Voila! now you have successfully connected your local Docker environment to Portainer and you should be able to see below screen:

7. Click on the local endpoint to see all the containers, images, volumes, networks etc. in your Docker environment:

8. You can also deploy app templates containers right from inside the Portainer:

This is it! Go on, play around a bit and I'm sure you'll love how easy Portainer makes it to manage Docker containers. Given how many of the self-hosted apps can be deployed using Docker containers, Portainer is a must-have tool in your arsenal.

In the LowEndSpirit, we tend to look for resource-efficient alternatives. Here is an alternative to use instead of Postfix, Sendmail, or Exim.

Often when installing and running a web application or script you need an SMTP server to send an email, rarely there is the need to receive any email. It works equally well using ssmtp, which also is simple and fast to install. It takes two minutes to install and configure.

RedHat, CentOS7, Fedora

yum install ssmtp

If you receive a “Package ssmtp is not available” error, you’ll need to install EPEL on your machine with the following command:

yum --enablerepo=extras install epel-release

Once done, you’ll be able to install ssmtp using the above command.

Ubuntu, Debian

apt-get install ssmtp

The configuration is done in the /etc/ssmtp/ssmtp.conf and there is only a couple of settings to change:

Mailhub
The mail server you must send mail through (relay). In this guide we will use GMail smtp Server.
From Line Override
Set to YES to allow the use of others choose from addresses other than the system itself.
AuthUser
The username or email adress on the account used to login to gmail.
AuthPass
The password for above account
UseSTARTTLS
Set to Yes to use TLS when connecting to the SMTP server.

## Config file for sSMTP sendmail
## The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=postmaster
# The place where the mail goes. The actual machine name is required no
# MX records are consulted. Commonly mailhosts are named mail.domain.com
mailhub=smtp.gmail.com:587
AuthUser=name@gmail.com
AuthPass=YourtopSecretPassw0rd!
UseSTARTTLS=YES 
# Where will the mail seem to come from?
#rewriteDomain= 
# The full hostname
hostname=debianVPS.local 
# Are users allowed to set their own From: address?
# YES - Allow the user to specify their own From: address
# NO - Use the system generated From: 
addressFromLineOverride=YES

No reboots required.

To use ssmtp with the PHP mail() function, you have to edit the sendmail_path parameter in php.ini to something like this:

sendmail_path = /usr/sbin/ssmtp -t

You have no open ports, everything just works!

Even if it is too late when someone else logs in as root on your server, it is good to know that NOW is the time to get working on your security.

To setup email notification, login to your server as root.
Edit .bashrc

editor .bashrc

add the following line at the end, changing “ServerName” to the hostname of your server and “email@thisaddress.com” to your own email address.

echo 'ALERT - Root Shell Access (ServerName) on:' `date` `who` | mail -s "Alert: Root Access from `who | cut -d"(" -f2 | cut -d")" -f1`" email@thisaddress.com 

!NOTE! - Take notice of the ` in the code block. Sometimes when copying code from a webpage, these small characters will mess up when pasted.

Save and exit your .bashrc.

Next time someone, hopefully you, logs on as root, you will get an email about this.

This can be done for any user, you only have to make sure that the user can email out from your server.

Quick Tip: Use the guide here to install and configure a Low Resource SMTP Server.

Following up on the bonus tip posted on Resize your KVM VPS disk partition, 2 methods and bonus tip to reclaim disk space – Easy mode, here is a longer explanation and guide how to reclaim your reserved disk space.

Joe Dougherty from SecureDragon.net (great guy running a great company) sent me a tip about this thread and asked if I could write something about this “weird trick”. Actually it’s not a wierd trick, it’s a built in security feature. The information in this post will only work on dedicated servers or Virtual Servers that utilize full virtualization, meaning that this won’t work on OpenVZ.

On a newly created filesystems (Ext [2/3/4]) some of the space will be allocated for the system superuser (root) as “system reserved”. The default of 5% is meant for system partitions. If something goes wrong and your server consumes all its free disk space, the root user could still log in and check logs/crashdumps/etc and generally fix the situation.

For example, if your disk space fills up, the system logs (/var/log) and root’s mailbox (/var/mail/root) can still receive important information. For a /home or general data storage partition, there’s no need to leave any space for root. For very special needs, you can even change the user that gets this emergency space.

There’s another reason to not allow an ext[23] filesystem to get full, which is fragmentation. Ext4 should be better at this, as explained by Linux filesystem developer/guru Theodore Ts’o:

If you set the reserved block count to zero, it won't affect performance much except if you run for long periods of time (with lots of file creates and deletes) while the filesystem is almost full (i.e., say above 95%), at which point you'll be subject to fragmentation problems.  Ext4's multi-block allocator is much more fragmentation resistant, because it tries much harder to find contiguous blocks, so even if you don't enable the other ext4 features, you'll see better results simply mounting an ext3 filesystem using ext4 before the filesystem gets completely full.If you are just using the filesystem for long-term archive, where files aren't changing very often (i.e., a huge mp3 or video store), it obviously won't matter.

Theodore Tso

If you have a VPS with small disk size the 5% won’t mean much but if you have a 100GB drive or bigger, it quickly adds up to a vaste amount of unused space. In those cases we could lower the amount of reserved space in order to claim and use a few more GB.

At the time of writing the original post, I actually had an unused XEN VPS so lets have a look at what we can do about this by using that as a real life example.

first we confirm the filesystem parameters by running this command:

# tune2fs -l /dev/xvda1

it will list all information about the disk. This is the output I got from my server:

tune2fs 1.42.5 (29-Jul-2012)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          50fd54e4-7740-4683-b1e5-64e93d6d1e92
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              9830400
Block count:              39321600
Reserved block count:     1966080
Free blocks:              38473681
Free inodes:              9799099
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1014
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stride:              1
RAID stripe width:        80
Filesystem created:       Mon Nov 10 19:05:08 2014
Last mount time:          Sun Dec 14 17:25:37 2014
Last write time:          Sun Dec 14 17:25:13 2014
Mount count:              12
Maximum mount count:      34
Last checked:             Mon Nov 10 19:05:08 2014
Check interval:           15552000 (6 months)
Next check after:         Sat May  9 19:05:08 2015
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      e2ccf267-28ea-4e34-9df0-a349d06f0247
Journal backup:           inode blocks

The ineresting part from the output above:

Reserved block count:     1966080
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

Before we move on to the amount of reserved space, take a moment to reflect on what user who is allowed to use the reserved space. By default it is root unless changed by the system administrator.

if you multiply the Reserved Block Count with the current Block Size (also found in the tune2fs output above)

Block size:               4096

we get how much space in bytes that is reserved by the system:

Doing the same operation using the Block Count value:

Block count:              39321600

will give you the Total Disk space of the drive

As you can see (7,5GB out of 150GB) exactly 5% of the disk is reserved space.

As previously mentioned, if you don’t have a large disk it would be wise to not change that 5% value since it could mean that you wont have enough “system reserved space” to recover from a full disk problem.

In my case, 7,5 GB of reserved space is a bit much and I would benefit if this was available for me to store my backups instead. So, how do we change the amount of reserved space?

Since my disk is in total 150GB each percentage is 1,5GB and I think that 1,5GB will be enough for this server, the command to set the reserved space to 1 percent would therefor look like this:

# tune2fs -m 1 /dev/xvda1

The returned result :

Setting reserved blocks percentage to 1% (393216 blocks)

Keeping in mind that each block is 4096 bytes the above result means the reserved space is:

393216 * 4096 = 1,5 GB

Before you jump of joy I would like to end this article with a few words of caution;

While this is a nice way to get some extra space on your server TAKE EXTREME CARE if you decide to change the settings on the drive that has the / volume or you could end up with a server that even root can’t save when the disk runs out of space. If you have a secondary drive that only holds data, may it be your mp3 collection or family photos, you can set the reserved space to 0percent on that drive. As long as it is NOT the system drive.

Introduction

In Part I of this series, we downloaded the Debian netinst install iso. We then created a KVM VPS with the iso attached, and, finally, we successfully booted the iso.

In today's post, we're going to install our KVM with Debian 10 from the newly booted iso. But first, a bit of context on installing.

Context

• Why the Debian minimal netinst iso?

Debian themselves say, "we think that in many cases the minimal CD image is better — above all, you only download the packages that you selected for installation on your machine. . . ."

What we gain from this series is a well-proven, widely used, minimal, highly extensible, open-source server operating system.

• What about networking?

The biggest difference between installing on our VPS and installing on our personal laptop or desktop might be network configuration. On personal devices, we are used to automatic network configuration happening behind the scenes via Dynamic Host Configuration Protocol (DHCP). We turn on our device, it gets its own IP address and internet connection without our having to do much.

On servers, however, the server's IP address and internet connection sometimes are set by hand instead of automatically via DHCP. Traditionally, server network settings are done from a console physically connected to the running server. Obviously, however, if our server is at a remote location, we cannot have a wired connection. Also, since networking hasn't yet been set up inside the server, we can't connect directly to our remote server over the internet, either.

As might be expected, the Debian minimal netinst iso is set up to configure networking automatically via DHCP. Thus, when we try the networking step of the install, that step will fail. The netinst iso will succeed, however, in installing a minimal Debian system without networking. In Part III of this series, covering Post Install Configuration, we will use the Proxmox web GUI and VNC to go inside our minimal system and set up networking by hand.

• Alternative installation methods

It might be worth mentioning a few of the many other excellent methods of server installation which, although frequently used, are not selected here because they might be even more complex than our "simple" method.

• First, Debian unattended Installation using a preseed file will not work here because no networking is set up to use for obtaining the preseed file.
• Cloud-init is "the industry standard multi-distribution method for cross-platform cloud instance initialization." However, the Proxmox Cloud-Init Support wiki article says, despite the convenience of ready-made images, "we usually recommended to prepare the images by yourself," because "you will know exactly what you have installed." Also, for a special perspective on Cloud-Init, you might enjoy watching Cloud-Init: The Good Parts.
• Proxmox supports Templates. It's possible to create templates with Packer. If interested, you can check Creating proxmox templates with packer.

Before We Start

We need to begin today at the exact stage where we left Part I. Our Debian Installer should be booted and running on our VPS.

We also will need the server's hostname (which can be Debian) plus the username (which also can be Debian) and the real name for the user account which the installer will create. It's also convenient to have on hand two previously generated good passwords, one for the root account and another for the new user account.

Debian Installer Steps

• Select Install

• Language

• Location

• Keyboard

• DHCP Tries and Fails

• Select "Do Not Configure Network at this Time"

• Hostname

• Enter and Confirm the Root Password

• User's Real Name

• Username

• User Password

• Time Zone

• Partitioning Method

• Disk to Partition

• Partitioning Scheme

• Confirm Partitioning

• Write Changes to Disks

• Confirm No Additional Install Media

• Confirm No Network Mirror

• Package Usage Survey

• Choose Additional Software

• Dual Boot

• Grub

• Installation Complete

In the Proxmox web GUI, we select VPS > Hardware > CD/DVD Drive. Press edit and select "Do not use any media." Then, we return to our "Installation Complete" screen by selecting Console, which should reappear just as we left it. Finally, we click the "Continue" button, which should reboot the VPS.

In Part I, we did not install Qemu Agent. Therefore, rebooting from the Proxmox web GUI (outside our VPS) as opposed to rebooting from the console (inside our VPS) might not work. However, if it is necessary to stop the server from the web GUI, we can use the web GUI's Stop command found on the drop-down menu of the Shutdown button.

• Successful Reboot

What is Axel?

When you want to download something from the command line you normally use the commands wget or curl.

What if you want to accelerate these downloads? I recently found this command: Axel which works the same way but allows you to use multiple connections for one file.
As a comparison, the Mozilla extension DownThemAll does the same thing in a graphical environment. Axel can also use multiple mirrors for a download and according to tests done by the Github author this can speed up downloads up to 60%!.

Axel has no dependencies, is lightweight, supports HTTP, HTTPS, FTP and FTPS protocols and unlike other similar programs, it downloads all the data directly to the destination file using a single thread., saving time at the end because the program does not ave to concatenate the downloaded parts. It is also available in the Debian repo

If you take a look at the options available, you see that you use it the same way as you would use wget.

Usage: axel [options] url1 [url2] [url...] 
--max-speed=x           -s x    Specify maximum speed (bytes per second)
--num-connections=x     -n x    Specify maximum number of connections
--output=f              -o f    Specify local output file
--search[=x]            -S [x]  Search for mirrors and download from x servers
--header=x              -H x    Add header string
--user-agent=x          -U x    Set user agent
--no-proxy              -N      Just don't use any proxy server
--quiet                 -q      Leave stdout alone
--verbose               -v      More status information
--alternate             -a      Alternate progress indicator
--help                  -h      This information
--version               -V      Version information

Axel in its simplest use:

axel url

Common options include

--max-speed

to prevent the app chewing up all your bandwidth and

--num-connections

to specify a number of connections (the recommended default of 4 is fine for most downloads).

Passing in multiple URLs allows downloading of the same file from multiple locations.

Summation

Axel is a great command-line tool, but what if you want a GUI download manager with similar features? Check out uGet, which also includes accelerated downloads, clipboard monitoring, browser integration queuing support, and many more features!

Download Manager for for Linux, BSD, Android, and Windows.

Following up on the post on how to loop thru a file and perform an action per line, which you can find here

https://lowendspirit.com/how-to-loop-through-a-file-and-perform-an-action-per-line/

There is a case when this is useful, adding IPs from a text file into iptables and block their access to your VPS or dedicated server.

if you break down this command with its parameters (iptables being the command)

iptables -A INPUT -s XXX.XXX.XXX.XXX -p udp -m udp --dport 28960:28965 -j DROP

Parameter: Explanation
-A: Append this to existing rules
INPUT: The chain where the rule should be added into
-s XXX.XXX.XXX.XXX: -s Sets the source for a particular packet, in this case the ip of XXX.XXX.XXX.XXX
-p udp: -p = Sets the IP protocol for the rule, which can be either icmp, tcp, udp, or all, to match every possible protocol. If this option is omitted when creating a rule, the all option is the default.
-m udp: -m = match option Different network protocols provide specialized matching options which may be set in specific ways to match a particular packet using that protocol. Of course, the protocol must first be specified in the iptables command, such as using -p tcp , to make the options for that protocol available.
–dport 28960:28965: –dport Specifies the destination port of the UDP packet, using the service name, port number, or range of port numbers. The –destination-port match option may be used instead of –dport. To specify a specific range of port numbers, separate the two numbers with a colon (:), such as our example. You may also use an exclamation point character (!) as a flag after the –dport option to tell iptables to match all packets which do not use that network service or port.
-j DROP: -j Tells iptables to jump to a particular target when a packet matches a particular rule. Valid targets to be used include the standard options, ACCEPT, DROP, QUEUE, and RETURN, as well as extended options that are available through modules loaded, such as LOG, MARK, and REJECT, among others. If no target is specified, the packet moves past the rule with no action taken. However, the counter for this rule is still increased by 1, as the packet matched the specified rule. in our example we use DROP — The system that sent the packet is not notified of the failure. The packet is simply removed from the rule checking the chain and discarded.

This command will DROP connections from IP XXX.XXX.XXX.XXX on udp port 28960:28965

If you want to block all connections from a specific IP, no matter what port it tries to connect to, omit the -p -m and --dport parameters. This will look like this

iptables -A INPUT -s XXX.XXX.XXX.XXX -j DROP

You might ask when are we going to loop thru the file?

#!/bin/sh

# This will loop thru the file /ban/banip.txt and add every IP in that 
# file with a DROP to the INPUT chain in iptables.
#
# change the path and file name if required

# you can re-run this file if you are not saving your iptables config 
# between reboots. 
while read blist
do
/sbin/iptables -A INPUT -s $blist -j DROP && sleep 2
echo $blist has been added to your iptables

done < /ban/banip.txt

To add a single IP to the block list in iptables and add the IP to your text file, you could use a simple shell script like this

#!/bin/sh
# Script to add ip
echo -n "Enter the IP to BAN and press [ENTER]:"
read ip
/sbin/iptables -A INPUT -s $ip -j DROP

#keep a record of the banned IP's if you want or comment out
echo $ip >> /ban/banip.txt
# Make sure you use the same path and filename as in the loop script

This is a quick and dirty way to keep a list of IPs you would like to block access from.

I'm sure that the readers have more sophisticated and innovative ways to add their own list of IPs to iptables.

Comment with how you do it and why you do it the way you do.

The main use case for this setup is to mix and extend storage options due to low local disk space, adding fast Nvme or slow cheap HDD for iso's,templates etc ...

While Proxmox cp does provide a GUI to add iscsi storage my experience with server restarts did not recover connectivity with the target disk.

In this tutorial i will walk though mostly a command line setup between two servers.

Note: It is recommend to use a private network between the two servers and if possible a 10GB connection would be ideal.

Terms to know

Target - "is the server providing the disk/partion to share"
Initiator - "the proxmox server consuming the target"

The next diagram depicts the main commands and files to touch:

are you ready; here we go:

A - Setting up the Target example ip - 98.76.54.32 :

# apt-get install tgt 
# cat <<EOF > /etc/tgt/conf.d/target01.conf
<target iqn.2021-03.kvm:lun1>
     backing-store /dev/mapper/vgstore-lviscsi
     initiator-address 12.34.56.78
     incominguser iscsi-user password
</target>
EOF
# systemctl --now enable tgt
# systemctl status tgt  ### make sure its running ; you may need to reboot

in the above conf file i am setting the free partition /dev/mapper/vgstore-lviscsi make sure you add the correct partition available to you.

# tgtadm -o show -m target

you may need to reboot to get tgtadm show tartget details.

B - Setting up the ** Initiator ** example ip - 12.34.56.78 :

# apt-get install open-iscsi 
# systemctl --now enable open-iscsi
# reboot # if next command
# iscsiadm -m discovery -t sendtargets -p 98.76.54.32:3260  # should get next sample output
98.76.54.32:3260,1 iqn.2021-03.kvm:lun1

next is to change the target conf file

# ## use tab after nodes to auto complete the name
nano /etc/iscsi/nodes/iqn.2021-03.kvm\:lun1/98.76.54.32\,3260\,1/default

find the key and edit value according to the following

node.startup = automatic
node.session.auth.authmethod = CHAP
node.session.auth.username = iscsi-user
node.session.auth.password = password

# systemctl restart open-iscsi 
# iscsiadm -m node --login     # should return successful 

now lets check if the partition is added

Now its time to create the volume and display it

# pvcreate /dev/sda
# vgcreate vgiscsi /dev/sda
# vgs

C - Add the vg group to proxmox

next create and use the new lvm for creating a container like

and its done.

This was my experince of extending storage on a proxmox using iscsi. I hope you will have fun as much as i did.

Refs:
https://www.howtoforge.com/tutorial/how-to-setup-iscsi-storage-server-on-ubuntu-2004-lts/
https://www.tecmint.com/setup-iscsi-target-and-initiator-on-debian-9/

Final notes:
I would like to mention @gleert from https://www.naranja.tech/ for providing a great kvm with a lot of bandwidth.
I promised him that if my install worked and ran smoothly for a while i will add this note.

Contributed by Not_Oles, December 27, 2020

Introduction

This afternoon we're going to spin up a new Virtual Private Server ("VPS") running the Debian GNU/Linux Operating System. Our VPS will be a Kernel-based Virtual Machine {"KVM") utilizing the Proxmox Virtual Environment.

Today's post will cover creation of the Virtual Machine ("VM") with the Proxmox web GUI ("Graphical User Environment") from the beginning up to the exciting moment when the Debian installer successfully starts. The next post will cover the Debian installation, and the following post will cover configuration inside the newly installed Debian operating system.

We're going to be using an AX51-NVME server at Hetzner on which Proxmox already has been installed.

Soyoustart, LXC, and Secure Shell Alternatives

If you do not already have a Proxmox server, previous posts covering Proxmox installation and postinstall configuration at Soyoustart might be helpful.

Today's project involves a KVM VPS. Another post in this series discusses creating LXC container VPSes.

Today's post also assumes a bit of familiarity with Secure Shell ("ssh"), which is used here to obtain the Debian netinstall ISO file. However, alternate instructions also are given for uploading the ISO via the Proxmox web GUI.

Download the ISO

For today's KVM VPS we're going to use the standard Debian netinstall iso.

• Log in to the server and put the Debian iso in the right place for the Proxmox web GUI to find it.

$ ssh root@hels.example.com
Linux hels.example.com 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Dec 13 06:26:17 2020 from 187.XXX.XXX.XXX
root@hels ~ # cd /var/lib/vz/template/iso
root@hels /var/lib/vz/template/iso # wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.7.0-amd64-netinst.iso
[ . . . ]
HTTP request sent, awaiting response... 200 OK
Length: 352321536 (336M) [application/x-iso9660-image]
Saving to: ‘debian-10.7.0-amd64-netinst.iso’

debian-10.7.0-amd64-netinst 100%[===========================================>] 336.00M   108MB/s    in 3.2s    

2020-12-15 23:34:41 (104 MB/s) - ‘debian-10.7.0-amd64-netinst.iso’ saved [352321536/352321536]

root@hels /var/lib/vz/template/iso # 

Verify the ISO

• Next we download the checksums file and the signature file so we can verify the ISO:

root@hels /var/lib/vz/template/iso # wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS
--2020-12-16 04:50:34--  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS
Resolving cdimage.debian.org (cdimage.debian.org)... 2001:6b0:19::165, 2001:6b0:19::173, 194.71.11.173, ...
Connecting to cdimage.debian.org (cdimage.debian.org)|2001:6b0:19::165|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 658
Saving to: ‘SHA512SUMS’

SHA512SUMS                  100%[===========================================>]     658  --.-KB/s    in 0s      

2020-12-16 04:50:35 (20.5 MB/s) - ‘SHA512SUMS’ saved [658/658]

root@hels /var/lib/vz/template/iso # wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS.sign
--2020-12-16 05:03:56--  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS.sign
Resolving cdimage.debian.org (cdimage.debian.org)... 2001:6b0:19::165, 2001:6b0:19::173, 194.71.11.165, ...
Connecting to cdimage.debian.org (cdimage.debian.org)|2001:6b0:19::165|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 833
Saving to: ‘SHA512SUMS.sign’

SHA512SUMS.sign             100%[===========================================>]     833  --.-KB/s    in 0s      

2020-12-16 05:03:56 (35.1 MB/s) - ‘SHA512SUMS.sign’ saved [833/833]

root@hels /var/lib/vz/template/iso # 

• Next we check the signature on the SHA512SUMS file:

root@hels /var/lib/vz/template/iso # gpg --verify SHA512SUMS.sign SHA512SUMS
gpg: Signature made Sun 06 Dec 2020 02:46:09 AM CET
gpg:                using RSA key DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg: Can't check signature: No public key
root@hels /var/lib/vz/template/iso # gpg --keyserver keyring.debian.org --recv DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key DA87E80D6294BE9B: public key "Debian CD signing key <debian-cd@lists.debian.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@hels /var/lib/vz/template/iso # gpg --verify SHA512SUMS.sign SHA512SUMS
gpg: Signature made Sun 06 Dec 2020 02:46:09 AM CET
gpg:                using RSA key DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg: Good signature from "Debian CD signing key <debian-cd@lists.debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B
root@hels /var/lib/vz/template/iso # 

• We verify that the downloaded install file matches the SHA512sum:

root@hels /var/lib/vz/template/iso # sha512sum -c SHA512SUMS 2>/dev/null | grep debian-10.7.0-amd64-netinst.iso 
debian-10.7.0-amd64-netinst.iso: OK
root@hels /var/lib/vz/template/iso # 

• Alternatively, we might already have the ISO downloaded and available locally, or it might be a custom ISO that we ourselves made. in these and similar cases, we can upload the ISO via the Proxmox web GUI.

Log in to the web GUI at https://[Node_Name].example.com:8006. Note that we must use https. The server will send an empty response if we use http.

In the upper left hand Server View column of the web GUI, we expand the Node_Name by clicking the almost invisible ">" to the left of the Node_Name. Then we click on "Local" and on "ISO Images." When we click on the "Upload" button above the list of images, a dialog box opens to start the upload.

Create the Virtual Machine

• In the upper right of the Proxmox web GUI, we click the "Create VM" button.

• The "General" Tab is the first tab in the "Create: Virtual Machine" dialog.

Proxmox assigns a VM ID number, by default beginning with 100, but we might prefer to use a custom numbering scheme.

By default, let's use the reverse DNS provided by Hetzner as the name of the server.

Click the "Next" button to continue to the OS tab.

• In the OS Tab, we select from the drop down the OS image we previously downloaded and verified. We also check to see that the OS type and kernel version are correct.

• The System Tab is next. In the System Tab, we get to choose a graphics card, running Qemu Agent, and the SCSI hard disk controller. Let's go with the defaults.

• In the Hard Disk Tab, let's increase the disk size allocation to 100 GB and otherwise go with the defaults.

• In the CPU Tab, let's give this user 8 cores.

• In the Memory Tab, let's give this user 8096 MiB.

• In the Network Tab we can go with the defaults.

• In the Confirm Tab, let's click "Start after created."

• Selecting the newly created VM and "Console" lets us see the successful Debian boot screen!

Conclusion

At this point, the VM has been created, and Debian server is ready to be installed inside the newly created VM.

Summary

In this post, I will discuss different ways to set up a WordPress website in under 10 US Dollars a year. Learn why I decided to take this approach, and how you can find frugal web hosting for an entry level blog or portfolio site.

Note

I had originally set out to discuss different ways to install WordPress. As I began writing it, it became evident that the LES readers might also be interested in learning about hosting a WordPress site in under 10 US dollars a year. We will cover that in Part II of this 2 post series.

What you can expect to read in Part I?

In this part, I have elaborated my reasons for doing so. For more discussions related to WordPress, you can visit the All Things WordPress Discussion in the LowEndSpirit Forum.

Introduction

During the recently concluded Black Friday/ Cyber Monday 2020 (BFCM2020) deals on LowEnd Spirit and elsewhere, you may have picked up a VPS or two or a Shared Hosting Plan here and there. Which may have left you wondering,
"What do I do with all these hosting plans?"

The more experienced hands may use their BFCM2020 purchase for specific projects such as Virtual Private Server for Plex server, a NAT for a VPN, shared hosting for parking domains, and so on.

In many cases, you, the buyer of these Black Friday deals would want to set up some soft of a website. Maybe you are not a beginner, but you might probably the ' go to ' person for friends, neighbours, relatives or significant others. They may often rely on for advise. Now lets' say one or more of them want to set up a website.

A website in 2021 ? You must be joking!

In spite of the barrage of Social Media sites and apps, the number of websites has actually increased the over the years For those interested in numbers, this site has some facts and figures to peruse. Some of the data and the sources in such surveys or statistics might need a second look, but such information does offer a perspective.

The relative convenience of setting up a website these days seems to be a contributing factor for the increasing number of websites. The convenience could include:

• Beginner friendly Hosting, either through SAAS providers like Squarespace and Wix, or
  via shared or managed hosting that uses one click installers like Softaculous and Installtron.

• Less complexity of setting up a website due to 'no code' solutions

• Cost of hosting, if we exclude the recent cPanel hike and its coming impact.
• Tools for backup and updates
• Ability to host in multiple languages

Rather than pondering on these factors further, let us move on to our main discussion!

WordPress Hosting, the Frugal Way

To keep things simple, I have structured this post in two segments. In the first part, I will provide a quick overview of the ecosystem for WordPress hosting, and some of the providers in this space. Lets call this an appetizer for the main topic of discussion.

In the Part II we will look at the different ways of setting up WordPress under different hosting configurations. This is the most important segment, where we will talk about how to host a WordPress site for under 10 US Dollars a year. Our aim is not to go cheap, but to do so in a frugal manner. If you are starting out today, or are migrating a low traffic website, this is the section you should pay close attention to. I will talk about issues like traffic, load testing, basic configurations (or the "stack" of themes and plugins used by me. Also find included some links, resources and takeaways from my testing.

The Mad Rush for WordPress Hosting

Depending on which source you look up, the marketshare of WordPress in the web hosting space is on the rise. The numbers quoted often are: "WordPress occupies 40 % marketshare of all websites" to "more than 50% of all websites that use a content management system, use WordPress." W3techs also reflects on this trend.

Without flying the flag on behalf of Automattic, the company behind WordPress, let us look at the Universe of WordPress Hosting. This is a complex system with hosts, plugin and theme developers, website designers, social media and security consultants, content specialists, marketers, e and woo commerce specialists, and so forth. WordPress is open source and basic hosting and some themes and plugins are available at little to no cost. But the professional services (content, search engine optimization, security, etc..) might cost you a lot. So is the case with managed or specialized hosting.

Our focus is on WordPress Hosting, so let us take a look at it closely.
This pricing for WordPress hosting is a very interesting area where you have WordPress VIP from automatic themselves, which exceeds US dollars 5,000 a month. In the mid range, there are pricing plans from a whole array of providers that can run into hundreds of US dollars a year.

If you look up managed WordPress hosting plans typically start at 10 to 15 US dollars a month at entry level, but the median price or most providers start their plans at 30 or 35 US dollars. Annually, the might run into excess of $300. If you look at the specifications that they provide, you would notice that the specifications or page views are not very high. Many of the providers offer similar set of features as this list shows .

This may include CDN or content delivery network, image optimization etc. The key reason for pricing is because of support. But evaluating support is akin to wading through tricky waters. In recent weeks I have spent quite a bit of time reading up on customer service issues and overall service levels of some of these providers. Siteground in particular seems to be the most talked about (and often maligned) provider, largely because of their 'bait and switch' pricing and recent decision to limit WordPress support requests.

Pricing plan for WP Optim, used for illustration

Not everyone needs Feature-Rich Hosting

A typical user may have a question like

"I am looking at hosting a simple website which will have about 50 posts and around 200 images. Do I really need to spend over 200 dollars a year on hosting?"

If such a question is posed in the webhosting groups on Facebook or other sites, you will receive a list of provider recommendations that may look like the below table.

Before answering a yes or no, to the above question, I usually ask a set of qualifier questions myself. Many of which seem logical (atleast to me). For purpose of our discussion,

• We will focus on a blog or image / portfolio site only.
• Deeply discounted shared hosting plans from the likes of Hostinger and EIG companies are excluded.

Sample Checklist of questions for WordPress Hosting selection

- What kind of site do you have? 
(i.e.) Is it a Blog, portfolio, membership site, store with woo commerce etc..? 
- Do you plan to use a Content Delivery Network (CDN)? Or, do you have one currently?
- If you are looking to move hosts, what are the current (shared) hosting specifications?
- Where is your hosting located, and where do the maximum visitors come from?
- Do you use a monitoring service like Hetrix tools or Uptime Robot to 
monitor the website uptime?
- What is the current hosting plan fees and what is your appetite for a higher plan/ upgrade?
- Are you comfortable with a DIY setup, or do you need handholding? (i.e for an existing site,
 do you need help with migration, or for a new site do you need help with setting up the site ? )

Keeping those questions in focus, we will ddress some of these qustions in the second part of this post. Till then. I leave you with the screenshot of a sample website creted for this blog post.

Blocksy theme on ClassicPress: giving a Frugal WordPress Website with a modern look.

Resources:
1. You may also want to read my accompanying post o NAll Things WordPress discussion in the LES Forum. Downside of Expensive WordPress Hosting Plans

2.Prices for "premium" or "Managed" WordPress Hosting Providers

For any feedback and suggestions about this post, please leave a comment below or contact @vyas in the LES forum. All screenshots are taken from websites of respective providers. All other images are created by A Vyas, December 2020.

Did you upgrade your VPS and it looks like the disk is still the same size? Did you migrate your VPS to another host using the method in part 1 of this easy mode series and the target disk was bigger so you want to make use of that extra space?

I will show you in a few simple methods below with examples of resize2fs and growpart and also the super simple point and click GParted.

In the following image we get a look at the disk and partition size before any changes have been made so we have a good starting point to work from, we can see it is a 10GB disk with a single partition /dev/vda1 being pretty much the full 10GB.

I then resized the disk up to 15GB in the backend as you can see in the next image, the physical disk is now 15GB however the partition /dev/vda1 is still only 10GB.

Starting with the GParted method we need to boot the VPS into a rescue mode that includes GParted, most hosts will offer some ISO with GParted included or just a plain GParted disk, I tend to use sysrescuecd as it usually has the relevant tool-set to complete most tasks.

Now you need to change the boot order and mount the ISO of choice (that includes GParted).

Boot order:

Mount ISO and reboot:

Clicking the reboot button in SolusVM is what re-writes your config file on the backend so that the ISO mounts and the boot order changes, in Virtualizor you need to 'Stop' then 'Start', reboot, for some reason does not update the config file so the ISO will not mount.

You will then be able to open a VNC session using the built-in HTML5 VNC viewer or a direct VNC connection using a desktop client.

If you used sysrescuecd you should see something similar to this:

Pick the default option and when the rescue mode drops you at a prompt, run: startx as shown in the next image:

Give it a minute or so and the graphical session should start and look like this:

You will see a shortcut to GParted on the bottom left, in other distros you may find it via the menu system.

After you launch GParted you will see a disk overview and you will see that extra 5GB of unused space:

Right-click on the first partition on the left labelled /dev/vda1 and select resize/move:

Then grab the furthest extent of the partition and drag it to the right and click the 'resize/move' button.

Before:

After:

Then click the tick to confirm the operation:

Confirm:

That should be it, the partition should now occupy all of the free space on the hard disk.

You can now shut down the rescue mode by unmounting the ISO and changing the boot order back:

Tip: Hit power off first then reboot, you will not risk any data while in rescue mode without mounted partitions so no need to wait for the full shutdown, this saves you 30 seconds or so.

Now when you log in to your VPS again you will see that the partition size has grown:

Congratulations you are all done!

But what if you feel like using GParted was cheating, more like 'Super-easy' mode and you want to feel like you learned what is actually happening rather than just knowing where to click?

I hear you!, let's do it again then in a slightly different way, once again in the backend, I have increased the disk size on the same VPS to 20GB (an extra 5GB) as seen in the image below.

Once again, get in to rescue mode or boot with the sysrescuecd iso, I won't repeat the same images as above, the process is identical however just get to the point whereby you get a prompt do not run startx.

Now running fdisk -l /dev/vda we can see that the disk vda is 20GB and the first partition vda1 is 15GB, to extend this run:

growpart /dev/vda 1 note the space between vda and 1

This (growpart) will not always automatically grow the filesystem but what it is useful for is forcing the kernel to register the new disk size as some older kernels and tools may fail with you run resize2fs with an error along the lines of "Nothing to do"

So next we will manually run resize2fs /dev/vda1

As you can see in the above image the increased partition size is verified with lsblk /dev/vda which will show the disk and the partition sizes.

Power off the VPS, change the boot order back to Hard disk first again and reboot.

Now when you log in to your actual OS you can verify that the partition has in fact expanded:

BONUS TIP

The ext4 filesystem most commonly used, certainly for Linux VPS templates anyway will as standard reserve 5% of your disk space in case you run out, it is like a buffer zone to prevent catastrophe.

It is a relic from ext3 which was released in 2001, at that time the average physical hard disk in use was between 10GB and 20GB.

These days disks even virtual ones are significantly bigger and you take very regular backups... right? so 5% is often really over the top.

You can reduce this to 0 however I seriously do not recommend that, but you can sometimes gain a good little chunk of disk space back by pushing the reserve down to 1%.

This can be done live, no need to reboot or go in to rescue mode, just run: tune2fs -m 1 /dev/vda1

The 1 is the % of space to reserve for the partition you select, in this case /dev/vda1

Contributed by @Not_Oles -- June 21, 2020

Introduction

Every interaction with a website requires a client (often a web browser) and also a server. Servers are computers on the internet which transmit content in response to requests sent by clients. The server which supplies the specific content responsive to specific requests often is called the host for that content. Thus, a hosting provider is a person or a company which provides servers that can host content.

OVH, SoYouStart, and Kimsufi

OVH, an international company based in France, is an example of a hosting provider. As of 2017, OVH had over 2,000 employees, 27 data centers worldwide, and over 300,000 active servers.

In addition to selling under its own name, OVH also has two additional brands: SoYouStart, which provides less expensive servers, and Kimsufi, which provides very low cost servers.

As appropriate in the world of Low End Spirit, we will be working here today with one of the lowest cost SoYouStart servers. Our server was rented as a great Boxing Day deal for just US$29.95 per month.

Here are our server's specs: SYS-LE-1 Server - Intel Xeon D-1521 - 32GB DDR4 ECC 2133MHz - 2x 2To HDD SATA Soft RAID. SoYouStart servers are advertised as 250 Mbps bandwidth, which is symmetrical and unmetered. However ssssh! a few SoYouStart servers seem to be being delivered with 500 Mbps or even 1 Gbps.

Avoiding waste of server resources

Even our inexpensive SoYouStart server is bigger and more powerful than what is needed for the majority of websites or web services. And a latest generation web server might offer ten or twenty times the power of ours. Thus, to avoid the waste of server capacity which would result if each website or web service each had to use its own, individual dedicated, bare metal server, many methods have been developed for sharing or "virtualizing" server capacity.

Proxmox Virtual Environment

Proxmox Virtual Environment ("PVE") is a free, open source server virtualization administrative system from an Austrian company. Proxmox has seen wide adoption. Proxmox software is designed to divide and share bare metal host servers and their resources among multiple virtual private servers (VPS). Proxmox VE 6.2 was released on May 12, 2020.

Proxmox offers a Graphical User Interface ("GUI") through which one can administer, via a web browser, Proxmox itself as well as Proxmox host servers ("nodes") and VPSes installed by Proxmox. Proxmox has an extensive wiki and an active forum. Proxmox paid support subscriptions are available.

Proxmox is programmed in the PERL ("Practical Extraction and Report Language") programming language. Because Proxmox is programmed in PERL, Proxmox VE users have, in addition to the GUI, the option of controlling everything via the traditional command line interface ("CLI").

The SoYouStart Control Panel

OVH is a Proxmox Hosting Partner, and Proxmox is available on the SoYouStart Control Panel for install as one among the many OVH supported operating systems.

Besides installing supported operating systems, and somewhat similar to the OVH and the Kimsufi control panels, the SoYouStart Control Panel includes facilities for:

• Rescue Mode -- booting the server into an entirely memory resident operating system
• Netboot -- booting the server's resident operating system on a kernel obtained over the network
• IPMI -- Intelligent Platform Management Interface, Supermicro's method of managing the server remotely, sometimes referred to as Keyboard, Video monitor, Mouse (KVM)
• Failover IP assignment and transfer -- add or remove additional IP addresses or transfer IP addresses from one server to another
• Setting Reverse DNS
• Real time server monitoring
• Backup
• Quite a bit more

One possible advantage of using Proxmox on OVH is that the Proxmox OVH install seems to use kernels from Proxmox instead of the OVH-customized, real time monitoring-enabled kernels present in the OVH-Control-Panel-installed versions of some other operating systems. On Proxmox, one thus gets newer kernels faster, while paying the price of losing OVH's real time monitoring in the OVH Control Panel. As of this writing, the Proxmox kernel version is SMP PVE 5.4.44-1 (Fri, 12 Jun 2020 08:18:46 +0200) x86_64, while the latest OVH version is 4.19.128.

Proxmox OS Install process

We're now going to run through the Control Panel's Proxmox install process,. Note that installing Proxmox is quite similar to installing any of the other supported Operating Systems. So, for example, if you wanted to install Debian or Ubuntu or CentOS, the install process looks very much the same. Indeed, Proxmox runs on Debian, and so a Proxmox install basically is an extended Debian install.

What really does look quite different, however, and lies beyond today's scope, is installing an unsupported OS like OpenBSD by using the Rescue System or the IPMI. Suffice it to say that almost anything compatible with the hardware should be installable and should work just fine, though getting it installed and working sometimes can be quite tricky.

Starting the Install

On a newly rented server the first view of the Control Panel shows a pop-up flagging the Reinstall button along with a message saying something like, "Your server is ready to be installed." On a server which already has had an install, we begin by clicking the Reinstall button in the upper right:

Default or Custom Install

The Installer presents us with an opportunity to select the default install or a custom install. The default install simplifies the whole processes by bypassing the opportunity to change the disk partitioning. The defaults are fine for many people, but maybe we will want to make a change, so, let's check the box for "Custom Installation:"

OS Template selection

Next we select the Proxmox VE 6 operating system template. Note that previous versions of Proxmox also are available further down the list. If we were installing another supported operating system besides Proxmox, this is the place where we would select it.

Disk RAID, partitioning, and formatting

At the disk partitioning and formatting stage, we have the opportunity, on servers with more than one disk, to use the disks redundantly. When one disk constantly is mirroring another disk in a Redundant Array of Independent Disks (RAID), we always have two copies of all our data. When one disk fails, our data is safe on the mirror disk. The cost of RAID, with two disks total, is a 50% reduction in disk capacity.

For production, many people, including me, would enable RAID. Thus, the default of the OVH installer is to have RAID Level 1 mirroring enabled. RAID 1 mirroring requires two disks.

But, here, for our non-production, test server, not using RAID 1 will double our disk capacity and also might permit us to test, on the second disk, additional partition types, file systems, or operating systems not available through the OVH installer. So, here, let's disable RAID 1 by clicking the checkbox for "Install on the first disk only."

If you want to keep RAID 1 enabled, simply leave the "Install on the first disk only" checkbox unchecked.

Next, we review the default partitioning scheme and make any desired changes. With First Disk Only selected the default partitioning scheme is as follows:

Here we consider two changes. First, changing the root partition ("/") to increase the size. The default 20,000 MB size is fine for the base system, and often, on production systems, one conservatively might want to avoid making any changes at all in the base system or its layout. But, for example, the base system does not include even the Proxmox manual pages. Since plenty of space is available, and since we might want to add a few things, let's increase the root partition size to 500,000 MB.

The 1024 MB swap space also might be fine, especially in a system, as here, with lots of RAM. Remember, we have 32 GB RAM in the host system. But probably we will not assign the VPSes that we will create with Proxmox access to the entire 32 GB memory. Therefore, a little more swap might help some of the smaller VPSes complete memory intensive projects without running out of memory. Let's set the swap to 16,384 MB, an amount about equal to half the total RAM.

Here's an image of the installer at the moment before confirmation of the resized root and swap:

Hostname, SSH key, and Post-install script

Next we tell the installer our fully qualified hostname. Also, we have an opportunity to select an ssh key from among the keys associated with our SoYouStart account. And, if we wish, we can link to a post-install script:

Confirmation

Finally we get our last chance to cancel or else to go ahead and launch the install:

Success

As soon as we confirm the install, all the magic begins to happen behind the scenes. On the Control Panel, the third entry in the left hand column, "OS," comes alive. It says something like "Initializing the install process," followed by "Hardware checks," followed by "Admin password." The install process continues, through "Partitioning and formatting," "Deploying OS," and "Rebooting." By this time, approximately ten minutes have passed.

The next and last stage takes about ten additional minutes for a total of twenty minutes all together. During the last stage, the Panel says, "Waiting for services to be up." Finally, we see a green "Success" pop-up, and also our phone beeps to announce the arrival of OVH's email confirming the install:

What's next after a successful install?

After our successful install we surely can congratulate ourselves, since now we're "(almost) ready" to begin serving profiting as full fledged Hosting Providers. Let's now take a quick look at what else might be needed.

Regarding our newly installed test server there remain at least: updating, upgrading, setting up the firewall, setting up fail2ban, downloading VPS OS templates, and actually creating VPSes. The next post will cover how to do these remaining steps with both the Proxmox web GUI and the command line.

In addition to the test server, we need enough additional RAID enabled production servers to provide the capacity needed by our customer base.

On the business administration side, if not already done, we need to establish our hosting company as a formal business under the laws of the jurisdiction which will be our home base. We need to build a website for our hosting business, add a payment gateway, and go on a marketing campaign unless we're already drowning in customer requests. Finally, we need to ask for our Hosting Provider tag so we can advertise on LES if we have space for additional customers.

This is not new information, I just wanted the wonderful LES community to be aware of this great time saver.

The official source is at https://wiki.debian.org/DebianInstaller/Preseed and a long file example at https://www.debian.org/releases/stable/example-preseed.txt

Requirements:

• A VM with Debian iso mounted e.g: Network install from a minimal CD
• The preseed file is uploaded in some webserver and reachable by the VM.
• The VM is online with vnc console support and is able to get an ip from your provider dhcp.
• The VM is booted from the iso, here are screen shots on how to get there:

Here is a working file "preseed-test.cfg" that partitions the entire first disk with lvm in auto mode.

#####  save this file as preseed-test.cfg  and upload to your webserver

## locals and keys
d-i debian-installer/locale string en_US
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select fi
d-i keymap select fi

## detect network-hostname and auto setup
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain

## mirrors and proxy if needed
d-i mirror/country string manual
d-i mirror/http/hostname string deb.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

## time and zone
d-i time/zone string Europe/Helsinki
d-i clock-setup/utc boolean true
d-i clock-setup/ntp boolean true
d-i clock-setup/ntp-server string pool.ntp.org

 ## Partion ::: use first disk , entire disk as one with lvm
d-i partman/early_command string debconf-set partman-auto/disk "$(list-devices disk | head -n1)"
d-i partman-auto/method string lvm
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman-lvm/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman/confirm boolean true

## add root and user with passwords, change later
d-i passwd/root-login boolean true
d-i passwd/root-password password startSimple20
d-i passwd/root-password-again password startSimple20
d-i passwd/user-fullname string anthonySmith
d-i passwd/username string ant
d-i passwd/user-password password startSimple20
d-i passwd/user-password-again password startSimple20
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false

## lets install a standard server with ssh
tasksel tasksel/first multiselect standard,ssh-server
# add your cool tools
d-i pkgsel/include string ntp ssh wget curl

## upgrades
unattended-upgrades unattended-upgrades/enable_auto_updates boolean false
d-i pkgsel/update-policy select none
popularity-contest popularity-contest/participate boolean false

## install grub
d-i grub-installer/bootdev  string default
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true

 ## eject and reboot
d-i cdrom-detect/eject boolean true
d-i finish-install/reboot_in_progress note

The installation will execute in auto mode and the result of the above file is shown next, ignore sdb I was testing with another disk.

Notes:
* I advise to read each section in the above preseed file and change accordingly to you needs for example; timezone, keyboard layout, passwords.... refer to references and source page.
* The partitioning can get tricky if you want to do a regular layout or exclude swap, but is doable.
* You can add custom scripts before install upgrade section like this as an example:

### my other scripts 
d-i preseed/late_command string \
   in-target sh -c 'sed -i "s/^#PermitRootLogin.*\$/PermitRootLogin yes/g" /etc/ssh/sshd_config'; \
   in-target curl -sq http://othersite.fi/k8s/99-k8s.conf -o /etc/sysctl.d/99-k8s.conf ;

What else have fun.

References

• https://www.debian.org/releases/buster/arm64/apbs02.en.html
• https://www.packer.io/guides/automatic-operating-system-installs/preseed_ubuntu
• https://gist.github.com/ofrzeta/afeb53590c538fbddace
• https://gist.github.com/lorin/5140029
• https://github.com/delfer/debian-preseed-iso/blob/master/preseed.cfg
• https://gist.github.com/boxrick/3a4022d003daa63b7d27cca7f0f99894
• https://gist.github.com/styblope/2cf93a41662608f924de71fd0e91e0d1
• others i might forgot to add.

Free Shared Web Hosting: Offers Galore

Over the past couple of months, providers on LES have generously offered free shared hosting plans for the LES community. The goal or objective behind is simple: they want to give back to the community and help those who can benefit from these offers.

In the below section I have compiled the list of offers by Four providers who have posted such offers since March 2020. Relevant details such as location, bandwidth, number of websites, disk space, are also included. This information is already available in the forums. However, wouldn't it be cool if it was available in a single page? I wanted to keep it simple, and while a more comprehensive tabulation like the one mentioned in this post might make sense for some, I opted for brevity and simplicity. With this thought, let us dive into these offers.

Presenting the offers Together

Notes:

1. DA stands for Direct Admin control panel
2. Free shared hosting plans by Hostingcubes are available for a period of one year at 7 different locations. Visit the offer page by @Lee - for further details
3. Requirements for Free Shared Hosting by Khan Web: "All you would need to do is join Discord Channel & send a PM to user @AK_KWH, mentioning your discord username"

The Obvious and Not-So-Obvious

There are several reasons to question the rationale behind such offers. For example, why would anybody offer a service for free? Then there is the suspicion that the providers might use the sign-ups as an opportunity to up-sell services. That may be true in some cases, but more than one provider in the above list has mentioned that these offers rarely result in paying customers. In the words of @seriesn from Nexusbytes,

"Usually if you are not paying for your project, you don't take it seriously logic applies here..... I guess it also depends because a lot of these use cases are for personal projects that never take off, or people just abandon them." - @seriesn

Risk of abuse by Users

The intention of this post is not to delve upon the pro's and cons of free shared web hosting. But I thought of mentioning that the providers can often face abuses from users and those looking to exploit the gratis service. This discussion from the forums, based on experiences of Lee with Hostingcubes, offers an interesting perspective.

Free service during beta testing

Some providers can offer free accounts for a limited period when they are launching new products or services. A recent example is Hello Internet /@hello, who offered free CPanel hosting in Germany. In exchange, they sought feedback, testing from the users. A provider may choose to continue offering the 'free' service to the beta testers beyond the testing period. 

Summing it all up

The act of offering a service for free does not limit itself to shared hosting on this forum. Some users have posted their offers for a 'free' use of VPS. But I thought of limiting this post to shared web hosting offers. One reason was focus. The other reason is more personal: I have benefited over the past years from free web hosting services. This prompted me to compile the offers and write this post that would provide a handy resource the LES community.

The details in the table are compiled from the offer posts. In some cases, I have used the details by checking with the providers, or from my own account. In case of any updates, errors, or additions, do leave your feedback in the comments section below.

Links to free shared web hosting offers by providers:

I have included them in the table above but for ready reference, here they are, in no particular order:

1. Tetahost https://talk.lowendspirit.com/discussion/1047
2. Nexusbytes: https://talk.lowendspirit.com/discussion/850/free-shared-hosting-servedez
3. Khan Web Hosting: https://talk.lowendspirit.com/discussion/772/free-directadmin-nvme-shared-hosting
4. Hostingcubes: https://talk.lowendspirit.com/discussion/636/lab-rats-required-to-test-new-free-hosting-service/p1

Contributed by @Not_Oles -- July 6, 2020

Introduction

Our goal is to become a Low End Hosting Provider by selling virtual private servers (VPSes) created with Proxmox on an inexpensive bare metal host server rented from SoYouStart.

In the first post in this series we successfully installed Proxmox Virtual Environment ("PVE"), Version 6.2 on a Low End SoYouStart server.

In this post, using the Proxmox web Graphical User Interface (GUI), as well as the traditional command line environment, we will accomplish postinstall steps, including:

• Initial Login,
• Obtaining a Let's Encrypt Secure Sockets Layer (SSL) certificate,
• Setting up the firewall,
• Adding firewall rules,
• Enabling firewall rules and also the firewall itself,
• Checking the firewall for proper operation,
• Updating and upgrading both Proxmox itself as well as the Debian GNU/Linux base system on which Proxmox 6.2 runs, and
• Additional ssh and web GUI security considerations.

The postinstall steps covered here are not specific to SoYouStart and, instead, are very much the same with other bare metal host server providers. So, this post could be used as a postinstall configuration guide for Proxmox installations on servers from other vendors besides SoYouStart.

Initial Login

Let's check the OVH email congratulating us on our successful install. Look within the email for "Application Access Parameters," which gives us a link to point our browser at the Proxmox web GUI and as well as our username, root, and root's password. Note that the link has "8006" in it because the web GUI runs on port 8006. If, using Chrome broswer, we click the link or paste it into our URL bar, the first thing we should see is the insecure connection block, which, looks like this:

Note that the connection actually is encrypted. It isn't really "insecure," except in the sense that Chrome does not recognize our server's certificate as valid. To access the Control Panel the block needs to be overridden, so let's click "Advanced," and then "Proceed to [the link from the email] (unsafe):"

We now should see the Proxmox web GUI login screen where we can enter root as our username and our password from the OVH emai and click "Login:"

Next we should see the Proxmox Nag popup:

Let's please consider buying a subscription to support the Proxmox team. Then we'll click OK.

Finally, we are ready to use the Proxmox web GUI.

The Control Panel layout features include:

• a Server View in the left hand column listing all the servers in the Datacenter. As this is our first server, it is the only server listed.
• The Top Bar, which tells us the version of Proxmox Virtual Environment, here 6.2-6, and provides a searchbox plus buttons to create a KVM VPS, create an LXC VPS, and to access root user account functions (Settings, Password, Two Factor Authentication (TFA), Language, and Logout).
• The main panel, which here contains information about the Datacenter, including a list of nodes (each server is called a "node") and a Help button in the upper right. The left hand column of the main panel contains a long list of categories, one of which is "Firewall," which we need to scroll down the menu in order to see.
• Recent Log entries are shown at the bottom of the main panel. We see that the zero currently existing VPSes were started without any failure.

We can click and drag the borders of the Control Panel layout sections to resize them as convenient.

Obtaining a Let's Encrypt SSL certificate

Ordinarily, the very first thing to do after installing a new server might be to secure the server by limiting access through the firewall, further adjusting ssh access, installing fail2ban, etc. However, these steps, as done below, do block Let's Encrypt's ability to issue the server certificate. To make issuing the certificate easier, let's get the certificate issued before setting up the firewall. Getting the certificate will only take a moment.

In order to get the certificate, you need to have registered a domain name such as superspeedyvps-example.com, pointed the domain's DNS records at the server's numerical IP address given in OVH's access email, and set up whatever email you want to use to receive notices from Let's Encrypt.

There is one additional preliminary tip to make using our new certificate easier after we get it. Initially we logged in to the Control Panel using the OVH supplied URL from the access letter. If, prior to certificate issuance, we login to the Control Panel using our own superspeedyvps-example.com domain, then issue the certificate, then login again, it seems to take Chrome a few days to stop putting up the unsecured connection block despite that Chrome seems immediately to recognize the newly issued certificate's validity. Therefore, let's briefly continue using the OVH supplied URL and not point Chrome at our superspeedyvps-example.com domain until after the Let's Encrypt certificate is issued.

In the left hand Server View column, let's click on our server's name and, in the center panel's left column, scroll down if necessary, and also click on "Certificates." We should see this:

Next, we need to add our account in order to issue a certificate via the Automated Certificate Management Environment (ACME) We click on "Add ACME Account."

Next, we add our email address, carefully review the Terms of Service (TOS), check the "Accept TOS" box, and click "Register."

We see the account registration output and also that the registration task succeeded.

Next we close the account registration output window and click "Add" in order to add our certificate. The Create Domain dialog box appears.

After clicking "Create," we click on our domain now newly appearing in the main panel's ACME domain list. Then, to get our certificate issued, we click on "Order Certificate Now." A pop up window appears, showing the output of the certificate ordering process. The process takes a minute or so, and ends with "Task OK."

The Chrome security block soon reappears because the certificate has changed. We can just close our tab and then reload the web GUI in a new tab, this time entering our own registered domain, superspeedyvps-example.com, into the browser URL bar. We have to log in again and we have another opportunity to see the Nag. We can click the lock icon in Chrome's URL bar and then "Certificate" to see our new certificate's details.

Setting Up the Firewall

The Proxmox firewall currently seems to use iptables-legacy. We can check our server this way:

# ls -al /etc/alternatives/iptables
lrwxrwxrwx 1 root root 25 Jul  2 19:40 /etc/alternatives/iptables -> /usr/sbin/iptables-legacy
#

The Proxmox firewall is disabled on a new install. When the firewall is enabled, the default firewall configuration is to block all but ports 8006 and 22 on the local network, so if we are connecting from the internet, we shouldn't enable the firewall itself until after wide area internet access rules are added and enabled. The Proxmox Firewall Instructions emphasize this warning:

The firewall is completely disabled by default, so you need to set the enable option [ . . . ]
Important If you enable the firewall, traffic to all hosts is blocked by default. Only exceptions is WebGUI(8006) and ssh(22) from your local network.
If you want to administrate your Proxmox VE hosts from remote, you need to create rules to allow traffic from those remote IPs to the web GUI (port 8006). You may also want to allow ssh (port 22), and maybe SPICE (port 3128).

Firewall changes do not affect connections previously made, so, as insurance, Proxmox advises:

Tip: Please open a SSH connection to one of your Proxmox VE hosts before enabling the firewall. That way you still have access to the host if something goes wrong.

Adding Firewall Rules

We are going to add firewall rules allowing web GUI and ssh access from our primary IP address and from a backup IP address so that our ability to use our primary IP address is not a single point of possible failure. We'll also add rules to allow the host to respond to ping and traceroute.

Here is how to add the rules using the Proxmox web GUI. First, we select our server in the Server View column. Second, we scroll down and select Firewall in the menu on the left side of the main panel:

Third, we click the "Add" button, and the Add Rule dialog appears:

The first rule is to accept incoming connections from our primary IP address on port 8006 and using TCP protocol. Let's not check the enable box at this time because, lest we get locked out, we want to add additional access rules before enabling. After entering each rule, let's click Add in the Add Rule box, then click Add on the Firewall main panel to relaunch the Add Rule dialog to enter our next rule. A list of all the rules we are adding looks like this:

If we are connecting from a dynamic IP, we can enter a rule allowing connections from a limited range of IP addresses using CIDR notation. For example, if only the last octet of our IP address changes, the source IP in the rule could be something like 123.123.123.0/24. An alternative to widening the connection rule is connecting from our dynamic IP through an intermediate fixed-IP VPS. Here, the IP of the intermediate VPS would entered into the rule.

After we have set all the rules, it's important for us carefully to triple check to make sure each rule is exactly, exactly, exactly right. A typo can lock us out! So, let's check all the rules one more time.

Enabling and disabling firewall rules and also the firewall itself

Proxmox allows us, as sometimes might be very convenient, quickly and easily to disable individual firewall rules, the entire firewall for an individual host server node, or even all the firewalls for the entire datacenter. Therefore, in order to enable our firewall rules, we need to enable each individual rule, the host server node firewall, and the Datacenter firewall.

First, we need to click the enable checkbox next to the beginning of each individual rule.

Second, we need to make sure that the host server node's firewall is on. Click on "Options," right under "Firewall" on the left column menu of the main panel. Verify that "Firewall," the first option shown on the list, is set to "Yes."

Third, to enable the firewall at the Datacenter level, we click on "Datacenter" in the far left Server View menu, then click on "Options" just under "Firewall" on the left column menu of the main panel. We need to change "Firewall," the first option shown on the list, from "No" to "Yes."

Now the firewall should be operational.

Checking the firewall for proper operation

Next, let's check to make sure the firewall actually is working as expected.

First, we can open a terminal on the host server node. Just being able to open the terminal shows that we are allowed access from our IP address. In the terminal, we can see whether each of our new rules appears when we run

# iptables -L PVEFW-HOST-IN

Second, we should try connecting from various IPs which should be disallowed as well as from the IPs for which we set specific ALLOW rules.

Updating and upgrading Proxmox and Debian

Our next step is to update and upgrade both Proxmox Virtual Environment and also the underlying Debian GNU/Linux operating system so that our customers' VPSes can benefit from the latest bug fixes and feature upgrades.

Like many things in Proxmox, the update / upgrade process can be done in a command line terminal window equally excellently as via the web GUI. Just open a terminal on the bare metal host server node and execute:

# apt-get update && apt-get dist-upgrade -y

In the web GUI, update plus dist-upgrade is a four click process. We click on

1. Our server's name in the extreme left Server View column,
2. "Updates" in the left column menu of the main panel,
3. "Refresh" on the top menu of the main panel, and
4. "Upgrade" on the top menu of the main panel.

"Refresh" on the top menu of the main panel launches a Task Window which runs apt-get update

"Upgrade" on the top menu of the main panel launches a command line terminal window which runs apt-get dist-upgrade.

Sometimes, following an upgrade, we are told to run

# apt autoremove

to take away a few old packages which no longer are needed. Also, I like to

# reboot

Rebooting may not be necessary after every upgrade, but I enjoy knowing that my server can reboot successfully.

Additional ssh and web GUI security considerations

Here are some additional security steps which we might consider. All these additional security steps have advantages and disadvantages, so each of us has to decide how best to proceed on each of our individual servers.

The first additional suggestion is to change the root password originally supplied by OVH. Most everyone would choose to do this. Additionally, we might consider disabling ssh password login entirely.

Another possibility is to change the ssh port from 22 to some high number. An alternative or further possible step is to use port knocking. An advantage of these is that the logs suddenly get very quiet, since most of script kiddie attackers do not pursue ssh after the port changes. A disadvantage is that many software packages and services assume ssh operates on port 22.

The ability to assume that ssh will be on port 22 is, after all, the purpose of having standard ports. Therefore, if we install port knocking or change the ssh port, some other things will break. For example, I no longer seem to be able to access my servers via the Intelligent Platform Management Interface (IPMI) console after changing the ssh port. Running multiple Proxmox servers from one server's web GUI might also break.

Yet another possible postinstall security step could be installing fail2ban. The default Debian fail2ban install protects ssh but requires additional configuration to protect the Proxmox web GUI. The default install also may require additional configuration to work with the Proxmox firewall. Here, where access to our server is limited to single IP addresses by our firewall ACCESS rules, fail2ban may be less necessary or even unneeded.

Conclusion

Following our successful Proxmox install in part one of this series and our successful postinstall configuration here in part two, we have reached the point where we can use Proxmox to create the VPSes for which our customers are eagerly waiting!

In the next post we will download an operating system template and then use the newly downloaded template to create our first VPS with Proxmox at SoYouStart!