Do you want to see who has downvoted you and who has upvoted you?

Or

Do you want downvotes to remain hidden?

To be clear, this is all or nothing; it includes historic votes as well, not just new ones.

This impacts:

• All Posts
• New member applications
• Hosting provider applications

I understand why this has come up a few times; transparency is important.

The flip side is, and the reason they were hidden to begin with, if you don't like something, you should feel free to express that without being singled out.

While I dont like the idea that the odd strange person can just downvote shit to get a kick out of it, the individual was never the point of this.

The point was so the collective community could self-moderate a lot of things like who gets banned, who gets a tag, who is just not fitting in well, which posts should be sunk and which ones are more visible.

It was supposed to be a collective power; I worry that revealing on a post who downvoted could be a barrier to some taking any action and make things generally more combative, but I also understand why people want to be able to see this.

I will go with the majority because, really, my own opinion is probably the least important here. I am steering the ship, but I did not pick your chosen destination.

If we do go for a full reveal, there will be rules to go with it; any arguments this causes with tit-for-tat stupid up/down vote wars will be squashed quickly and will come under the mod/admin time vacuum rules.

I think because the implications of this are pretty significant, I would like to say that the winning answer needs at least 50 votes to pass and must be at least 60% of the total votes; there is no throwaway I don't care vote here, and there will be no modifications after.

I removed the support link as there have been no support tickets for 30+ days now that most things that caused support tickets are automated.

This is the first of many posts in the new LES Blog, its tied in to vanilla so you dont need separate accounts to comment etc.

@mikho kicking things off with a series on the history of LE*/LES and the aptly named "Deadpool Files"

Anyway, there will not be a forum post for every new blog article so as not to flood the forum; instead, just a little counter you should see for any new/unread blog posts.

From that thread, a challenge was accepted in the cest-pit ... and a new thread were created about this project. https://lowendspirit.com/discussion/11268/vibe-coding-a-vestacp-alternative

That thread is the discussion thread about this project. This thread is only for the documentation and story abou the project.
As one of the main things said in the beginning was to keep manual entries at a minimum, most of the underlaying sources are created by Claude Code.

After each Plan phase and each Code phase, updates are timestamped and posted in a Decision.log.md file. That is the main source.

At first the plan was also to release this as seperate blog posts, but already, a couple of days into the project, we already have 10 parts, each part is aprox. a 10 minute read.

So this thread was created (and closed), to have everything without comments cluttering up the thread.
If you have a comment about it, use the thread mentioned above.

Basics

• Upvote a thread or post
• Downvote a thread or post

The cost of a basic upvote/downvote to you is essentially 0; it's just expressing your opinion
The cost to the OP or commenter is that they get +1 or lose -1

Good Karma can be used in multiple ways here and with more to come: revealing gated offers, the karma store and soon to buy promotion squares and to give to others that make some effort on any post or thread.

If a thread gets sufficient downvotes, it auto-sinks; if it continues to get downvotes, it gets hidden.

This essentially allows the community to self-moderate what they see; if you collectively don't like something, downvote it, please, as I have started doing on the obviously AI-generated posts.

User Impacts

• if a user is at 0 karma, they have a pleb badge and a grey avatar and name, a visual representation that they are not activley involved in much or new.
• if a user gets to -10 they get auto wanred to improve
• if a user gets to -25 they are automatically banned.

The community essentially manages who is here and what they want to see.

Dynamics:

• Up/down vote of OP weight = 1
• Up/down vote of comment on post weight = 0.25

This, for those who have the full karma animations enabled, is what powers this and how bright things get and how the post is coloured and highlighted; it also impacts which threads show up in the right-hand menu.

Again, this is all part of self modderation, its much quicker to downvote shit than to report it, it makes life easier for everyone, and everything is based on group opinions, not single ones, both positive and negative.

It is the best tool to actively steer the collective community direction; the power is in your hands.

I hope that clarifies. I have been meaning to post this for a while, but after a number of PM's and reports recently it is apparent that a large number of people have missed or skipped the original explanation (in fairness, that thread was trashed with bug reports)

Further to the previous discussion https://lowendspirit.com/discussion/11180/les-update-where-we-are-what-is-next-and-more-changes I am forking the advertising side of that discussion into this thread.

The support was good, and the overall feedback was positive, which, given that we are talking about the A word, is a bit of a surprise but shows good support for making this place grow.

What I have done.

I have plugged the karma system and FastSpring (card, PayPal, Venmo, Google Pay, Apple Pay, etc) into an advertising square system on LES.

It is a bit different to how it is done on LET who use BuySellAds, they only allow hosting related adverts and only allow adverts in general, they charge $125 per 125x125 square, LES who has probably only around 20% of the traffic is charging $7 per 150x150 square, which works out at about 2 - 3% of the cost per pixel for the same amount of traffic

We are also allowing any members to take a square, and on top of that, you can use your good karma to pay for them instead of money. If you have been active here and contributed well, then you have probably built up a lot of good karma (thanks), and now you can spend it on this, the price is 70 karma per month per square.

It is also different as you can buy a 1x1 a 2x1 a 1x2 or a 2x2 square, or as many of any configuration as you want, if you have a 300x300 image to show or space to use, you can take a 2x2 and you can add or remove as many as you want at any time and your subscription will update accordingly.

When is this happening

I am just waiting for the final conversion to go live from the FastSpring rep, so I expect probably at the end of this week at the latest, its just the flip of a switch on the backend now, and it goes live, and the link shows up.

What is allowed

Well, its for anyone, so you can advertise your hosting or services, you can advertise your kite shop, your personal website, another hosting related forum, your reddit group, your facebook page, really I dont care, the only rule is that it must be suitable for all age groups: nothing offensive, political, or anything trying to skirt a grey line "an adgitation" so to speak.

If you abuse it, it's a direct ban: zero warning, zero appeal, just gone along with the advert and your account, if you are unsure if something is ok you can always ask; no excuses accepted.

How it works.

You have a few choices: first, you select your square size and how you want to pay (FastSpring or Karma)

Then you can either upload your image (max frames = 2 hard cap) or you can upload multiple images and set unique links for A/B testing OR you can simply set the square not to display i.e. remove that square from the site.

You will get stats on how many impressions you get and how many clicks etc, it will also be broken down into what we can pretty much guarantee are human impressions vs 'could be bots, could be some sort of privacy extension, unsure if human though' rather than pretend the numbers are sky high by counting LLM page crawlers like other places do to inflate value.

With karma, you can book the square for up to 3 months in advance; you need to manually renew within 14 days of expiry to keep it.

With direct payments, you set up a subscription which you can manage from here and cancel or increase at any time, that is taken monthly on the date you set it up recurring, when possible there will be a pro rated cost if you add a square on the 1st and then want another on the 10th for example, if the fee is lessthan $2 then that is charged as a minimum to cover fees and to allow all endpoints to accept the transaction.

Squares are shown on the side menu only on desktop, and on mobile a 2x2 block (can be made up of 4 1x1 or 2 2x1 or just a single 2x2, every 7 posts or at the end of a thread with <7 posts per page.

Tax and VAT invoices and Payments in general.

FastSpring is a merchant of record; you are never a direct customer of lowendspirit, you are a customer of FastSpring, and FastSpring is our only customer, so we never need to store your personal information here at lowendspirit.

FastSpring will charge you any applicable VAT or local/state tax due at the point of transaction, and that tax will be paid directly to your EU member state or local state government office by FastSpring, all valid exemption numbers are accepted, if no tax ro VAT is due, none will be charged, if you use a VPN during payment, you will likley be either treated as a member of the origin state of your IP or rejected if you flag a risk factor for completing a purchae via a datacenter registered IP locatioed far away from your payment method.

Obviously no tax on karma.

Can I block it?

Probably and trivially, and I am genuinely not concerned if people do; it's just not worth worrying about. If you do not want to see adverts or content or links or pictures of pants or links to people's projects, no problem; that is your choice. I am not going to provide an on/off button, though.

Screenshots so you know what to expect:

How will this benefit LES.

• It will cover and recoup the costs mentioned in the previous thread
• it will allow the LES infrastructure to expand and become more redundant with a better backup system
• it will allow me to start hunting for VPS servers to review and bring new content into the community, not just from the regulars.
• I will be able to start doing bulk deals with hosts to buy down offers, which will be exclusive to LES.
• I will be able to do more work to try to advance some of the free service providers that post here to make life better for them and have them provide more resources to LES members.

Finally

I understand not everyone will like this; sorry, it is going to happen anyway, as this is the best idea I have to advance the community as a whole. If you have better ones, you have also had a lot of opportunity to present them; for now, try to live with it. If you can't, I understand.

If this fails, no one is interested, or it just ends up being a giant long-term karma-based block to hide all squares, I also understand. I am not going to try to fight it; it is what it is. I will take that as a sign.

I am still open to any other feedback and ideas, as always. Angry throwaway comments that add nothing of substance are generally mentally filtered out now, though, so feel free to throw them at me too

It's a low-hanging fruit kind of day.

Old unused roles and vanity roles have been removed; some other vanity roles have been kept until I get time to convert them into 1 of 1 badges in the karma store instead of membership groups, which should tidy up the massive bars a bit and make life easier for future updates and security considerations.

You should now see a quote button on mobile; you might need a hard refresh.

Should make life easier, I know I have just not bothered sometimes because it is such a pain.

Untested on IOS, any feedback would be welcome.

The top hosts this month metric is not working as expected, this has been disabled now and will be worked in to what shows in the side bar in another way.

Host username removed and items set to scroll the title on mouseover instead of basic truncation.

This is a stream from my head, this post is overdue so I will write it raw as it comes to me and get the jumbled mess out of my head.

This topic is not going to get a consensus; I already know this and accept it, some people want everything to remain the same; some want only the changes they want; some want to donate in some way, some want to donate in other ways, some dont see why this place needs funding; some don't care about adverts; some want specific types of adverts; and insert 20 other versions of all of those opinions and then dome just dont care either way.

That makes it literally impossible to please everyone, so I am just accepting I cannot please everyone, but I am being as sensitive as possible to all of this while also acknowledging that some people are going to feel more strongly than others.

The original idea was to use google ads and allow people to opt in, and always show them to none logged in users.

The issue with that:

• People opted in and then complained anyway.
• Google is a horrible system to use, has massive overreach and is overcomplicated for this platform in terms of compliance.
• For the last X years, LES has had no statistics running at all, so I had no idea if it was going to yield $0.01 or $1000 per month so I found everything out at the same time.
• Frankly, I don't like having Google integrated in anyway; I am really going off Google as a company in general. I don't think I am alone in this.

The current situation:

• I have managed to reduce the "lights on" costs to around $10 per month for a 2GB OVH VPS on top of that there is domain and email costs so lets call it around $150 per year, not alot really in the scheme of things.
• LES is now stable in terms of speed dips, backed up regularly (provider managed) , all possible caching is now activated at multiple levels and there is even a dedicated worker to catch all the AI scraper traffic so it never impacts the main forum speed priority.
• Vanilla has been brought up to date for php 8.4 and is running on mariadb 11.x and on debian thanks.
• I have done a few rounds of feedback before I touched anything, I appreciate sometimes people dont know what they want until they know what they dont want, and I did what I thought was a balanced mix of things to improve the flow of the forum, give the community more powers of self modderation and got you more involved in decision making while letting you have an active role in long term direction thought down voting, up voting that makes a difference, you can ignore pople, I have reduced friction on new members joining and hosts getting badges.
• The price limits are still removed, and will be reviewed early 2027, I just feel that in 2026 its too arbitrary and to exclusionary to set $7 limits.
• now I have had stats running on LES for a while via CloudFlare, the numbers are not great, from memory, around 7 years ago we were at around 300k - 500k page views per month. Today that's closer to 170k

What it took to get here and what is next in the short term:

• I have probably spent about $500 of my own money and put in around 200 hours of my own time bringing things to where we are now.
• The blog is coming back, some of you will remember the glory days of the LET front page, I have adapted vanilla to essentially allow WordPress-style blogposts while still using the vanilla engine, so https://lowendspirit.com/blog will start getting updates soon. I will likley be taking full personal control over that and it will be fairly active.
• I will be adding squares to the right-hand menu, there will never be agreement, adblockers exist, everyone can make their own choice. I will be doing it differently, everything will be managed in house and via the forum, similar to the karma store, you will be able to buy any amount of squares, you can use them for anything you want (anything appropriate) for 70 karma or $7, month, and you can combine them into a maximum of 2x2, other options will be 1x1 2x1 1x2 and you can use karma or pay to have them blank. Squares are open to all members not just hosts, Karma points will be accepted as payment as that means you have been of value to the community and forum, static images, 2 frames maximum, no flashing.
• Small Graphical signatures will be available (smaller than the current 3 lines of text) as alt images for mobile view.
• I have a ton of low-hanging fruit in terms of optimisation to do both on the cloudflare side and the general backend side to increase pagespeed scores etc to make the entire forum more appealing to search engines etc so I will work on that often.
• I need to continue to look for opportunities to evolve the offer here.
• I will be starting to remove providers, so we have a more accurate picture. I will do a mass PM to all hosts, and give them 30 days to login and explain to new push/lease of life/situation, any that don't log in will become regular members again.
• I am looking into adding more product/search capability for hosts for the providers section, something like serverhunter but not stepping on serverhunter's toes, although frankly I am not sure that is really much use these days it has been abused, and results are never accurate, frustratingly.

Why do you need any money?:

• A lot of reasons, not limited to, redundancy, offsite backups, my time is not free to me, never mind in general, stability and being able to make a better offer to the users, paid content, opening up opportunities to evolve the LES offer.
• More immediately, I want to start literally buying down offers for the community; I have already spoken to a few hosts about this, I want to be able to buy down hardware costs up front in various ways so that they can make better offers for you, ensuring LES users get 64GB ram instead of 32GB Ram or massive group buy discounts that are backed and guranteed.
• I want to start a good host scheme, something other hosts and I have talked about for a very long time, so people have a trust mark of sorts that has some accountability attached to it; this won't be free or cheap to put in place.
• I need to get more active members in here; frankly, the numbers are not on a healthy trend I see this as a last ditch effort.
• I want to help fund some of the free offers that are already here and increase that e.g. paying some of @Neoon 's and @msatt invoices for more exotic locations etc which then comes back to the community.
• I dont need a lot, but a working budget would be good.

What is the long term?

• That very much depends on how things go over the next year or so for LES, I promise I am going to continue to try as hard as I can to push this place, I appreciate some people just think it is fine how it is, but how it is, is going down in all metrics, so if you are one of those people perhaps you do not understand that what you are saying is essentially "It's fine, let it die slowly"
• My health, I have an appointment in 5 days that will dictate how the rest of my year goes, I did say and take on board the suggestion I should not hold all the keys and agreed to keeping a adminsitrator in the loop here in case anything happens to me; it's well above a 0 chance for me at this stage that I dont get better. with that said (sorry he is genuinley not expecting this) I would like to ask @mikho if he would consider being the saftey net and be an admin again, simply because he knows everything he needs to know, we have known each other well for 15+ years now and I would trust him with my private keys, if I vanish or just take a massive health dip I know he will do what is needed, if @mikho is not interested I understand I will make the offer open to all current staff roles.
• I do not plan to stick around as an administartor/owner long term, I will do what I need to do to get things turned around and step back, or reach a point where I accept I am not hitting the goals I need to hit and maybe I am not the right person, and I will step aside and let someone else try, likely from the existing Mod/Admin team.

So that's everything I can think of right now. Feel free to ask any questions; please accept this in the good spirit it is intended; writing is not my strong point, apparently, nor is getting my intent across a lot of the time, so please don't read between lines; you won't find anything.

This is what it looks like:

Options once enabled:

What the post looks like:

So you can use thresholds for members and be sure certain members only see your posts or sections of your post.

If anyone can think of a reason to have this option in other sections, please let me know.

There will now be 1 new members mega thread all new member posts will go in there not seperate posts.

New members (working on permissions) will have more rights to interact with existing threads outside of the "new members" member group, but will still not be able to see offers etc until on +5 karma

No one with less than 25 karma can give a new member an upvote and upvotes for new members can be given on any thread they interact with.

No one can gift karma to a new member; only up/down votes will work.

If users in the 'new members' group get -5 they are banned.

Participation from new members after registration is not mandatory but encouraged and rewarded; obvious spammers will be auto-removed quickly by the community downvotes.

Standard members are welcome to ignore the megathread if they have no interest; obviously, it helps LES in general to bring new members in.

I believe that solves all problems, or at least it's worth seeing how it goes before any further feedback is needed, but if you can spot an obvious problem that is not equivalent to "What if an asteroid strikes at 16:38 on a Tuesday"

If you want any other significant change related to this beyond tweaks, then please use the feature requests thread

Cheers.

Ant.

I am going to change it so that no one even sees the thread to vote on until they have made an actual post doing and introduction, I will make that a clear requirement.

If you are going to suggest changes or changes to numbers and requirements, please provide justification, like if not 5, why not 5. Nothing needs to be war and peace just give me a clue why you think what you think.

As a reminder the previous system was 1 question and 1 person deciding, it's just more visible now.

And the purpose was to reduce friction for new members to join and actually participate.

You just need to have at least 10 karma and the source file can be upto 5mb

You can add alt tags too if you want some SEO juice on your post.

Either use the image upload button, copy and paste an image or just drag and drop it into the text input and the forum will grab it and convert it and optimise it and strip exit data etc.

Going to implement an Imgur proxy in the coming weeks too that grabs and converts the Imgur URLs.

All of this is in the dev version not here yet.

The support ticket system has been replaced, it is built into vanilla now.

When you open tickets, all mods/admins get a notification, you could get an answer from anyone.

Final tweaks done to the new themes, and now the framework for adding new themes is finished, which gives a lot more flexibility than the standard vanilla system, so I might do a phpBB-style view at some point, too.

Link sharing is almost done (Hacker News Style)

New user gate almost done, new users will have to do an intro post etc before they can see or access certain areas. I am making this deliciously tempting for spam bots. Negative karma in here from older accounts will auto-ban the users.

Cloudflare Turnstyle is replacing the user registration manual approval system.

I am targeting the end of next week to get V2 live.

Turns out I've been working (for pay) with computers for over 30 years now, and I've tried and learned a lot during that time (still learning, it's a dynamic field). Here's a short-list of what's worked for me so far (in an infrastructure-related order so to speak):

For those who know my style - the series is mostly technical, brief, and to-the-point. That's what it is: basically a list with brief explanations why. Not my usual long-windedness.

My Hosting stack
Infrastructure - domain registration, DNS, website and email hosting.

My (backups) Storage stack
Storage - physical and cloud.

My OS & System Software stack
Computer-running - OS, drivers, security, diagnostics, and utilities.

My Application Software stack
Computer-using - writing, editing, playing media, and troubleshooting.

Writing this series got me to revisit and rethink my stack (helping myself too), and took me on a few trips down memory lane (like first time seeing Total Commander on my late uncle's PC, while I was just learning and using DOS command prompt at the time and similar).

It was fun, and, as the modern managers and HR folks say: challenging. LOL

Relja

P.S.
I think this topic logically belongs in this section, but if it's a bad idea, moving is one-click away.

I think I want a really peaceful, retro green terminal like the above screenshot.

I recently installed and wrote about vscodium and code-server, but vscodium, although super great, is not retro green.

In the retro green terminal, there is a peaceful, non-distracting, empty space into which I can load whatever I am working on at the moment. Any one of several editors plus a vast array of command line tools can be used instantly.

In addition to the command line tools, dot files and dot directories can be used keep everything arranged, close by, yet out of the way. Right now I use one dot file and two dot directories.

My dot file is .tlog, which is a simple text file with dates, log entries, comments, and terminal commands transcribed along with their output.

My two dot directories are .current and .attic. The first, .current, is where I keep individual files and project directories on which I currently am working, although not at the precise present moment. The second, .attic, is where I keep files and projects on which I am not currently working, but which I think I might need or want to see again before too long.

Once a new project hits .current or moves from .current to .attic, I still can go work on it there. Its not like I have to move the project back to my home directory in order to edit a file.

As a simple workflow example, here is me working on this post, comments added.

root@tx:~# su - tom # Logging in as root saves me from typing sudo a million times as day. 
tom@tx:~$ date # I need to remind myself of the date in order to add the date to the log.
Mon Oct 27 05:16:31 PM UTC 2025
tom@tx:~$ vi .tlog # Starting todays log entries. 
tom@tx:~$ cp -p .tlog .tlog~ # Local quick backup, just in case.
tom@tx:~$ vi How-do-you-set-up-your-terminal # Write todays post.j 
tom@tx:~$ ls # Does the post exist in my home directory?
How-do-you-set-up-your-terminal # Yes!
tom@tx:~$ vi .tlog # Add log entry for this post.
tom@tx:~$ ls .current/ # Not much yet on the new Texas server.
tom@tx:~$ mv How-do-you-set-up-your-terminal .current/ # Move the draft to the .current directory.
tom@tx:~$ clear # Restore the peacefulness of the clean view of my home directory.

The green color I am using is #00ED00, which reminds me of the IBM 5151 that I used to use.

For backup, I mirror my /home/tom directory on another server from a different provider. I can log in to either mirror and begin working right away. In addition to mirroring, I keep tar backups at a third provider and also on a local disk.

The biggest disadvantage of my retro green setup might be that it is stuck in time. I loved and lost the infinite scrollback from the slightly earlier days of paper terminals. Also, perhaps my love of retro green keeps me from embracing many advantages of newer terminal setups?

How do you set up your terminal? Can you convince me, almost 78 years old now, to move forward in time, to become young again?

code-server provides the ability to "[r]un VS Code on any machine anywhere and access it in the browser."

The vscodium README sends us to the @paulcarroty repo at Gitlab where there are specific Debian apt install instructions.

Install vscodium and/or vscodium-insiders

The specific install commands can be copied directly from the @paulcarroty repo. Basically there are only four simple steps:

• Install a gpg key;

• add the repository to /etc/apt/sources.list.d/;

• update, upgrade; and then

• install codium and/or codium-insiders.

Install code-server

Next is to install code-server. The code-server repo includes an install script, as well as the curl commands to test and run the install script.

Connecting

To connect, set up an ssh tunnel to the VPS from the local machine. Then, on the VPS, start codium or codium-insiders followed by code-server. Finally, go to http://localhost:8080 in the local machine's browser.

On the local machine:

• ssh tom@xxx.xxx.xxx.xxx -L 8080:localhost:8080

On the VPS:

• $ cat /home/tom/.config/code-server/config.yaml # Get the code-server password

• $ codium # Start vscodium.

• $ code-server # Start code-server.

Back on the local machine:

• Go to http://localhost:8080 in the browser.

Screenshots

The new FOSSBilling free and open source billing platform was discussed here at LES in December 2022. The 2022 discussion included very helpful background posts from FOSSBilling developers @jaapmarcus and @evrifaessa.

FOSSBilling has progressed a lot since the December 2022 LES discussion. Indeed, shared hosting providers already have been using FOSSBilling in production.

Recently, I signed up as a customer at two shared hosting providers who use FOSSBilling. This post details the results of my testing FOSSBilling from the perspective of a shared hosting customer. Also presented after the test results are, courtesy of @BelleNottelling, insights about the upcoming 0.6.0 FOSSBilling release along with tips for using FOSSBilling.

FOSSBilling Demo

The FOSSBilling website has a great demo, which enables everyone to see both the administrator panel interface and the client area interface.

FOSSBilling vs. Blesta

When FreeVPS.org was started, the FreeVPS Team went with Blesta instead of FOSSBilling because the FreeVPS Team had experience with Blesta. However, unless, like FreeVPS, you have a generous donor of a Blesta license (Thanks @Cloudie!), using Blesta normally requires monthly payments or more expensive lifetime licenses. Also, Blesta is not 100% open source.

FOSSBilling Tested At HostByBelle And At YetiNode

Shared hosting providers already using FOSSBilling in production include HostByBelle and YetiNode. Belle is one of the FOSSBilling developers. She is @BelleNottelling here on LES. YetiNode is run by our very own indomitable Yeti, @AuroraZero.

My overall experience testing FOSSBilling as a customer was excellent at both HostByBelle and at YetiNode. I had no trouble launching a test html page on each provider's shared hosting service. Here are links to the two test pages: HostByBelle and YetiNode.

Both providers use HestiaCP to provision shared hosting. Billing with FOSSBilling and provisioning with HestiaCP were very similar at HostByBelle and at YetiNode. I tried FOSSBilling's Support Ticket interface at HostByBelle. @AuroraZero talked with me by PM here at LES.

Screenshots Of FOSSBilling At HostByBelle

The gorgeous dark theme shown in these screenshots comes from the reverse colors engineered by Chrome Browser's High Contrast Accessibility Extension. These screenshots are what I actually saw during sign-up at HostByBelle. If you also wish to see the normal, non-reversed color mode, please look at the above linked FOSSBilling demo.

• Sign-Up

• Dashboard

• Ordering

• Payment

• Hestia Control Panel

• Invoices

• Support Tickets

• Quick Ticket Reply

Additional FOSSBilling Discussion

• Emails

FOSSBilling sent transactional emails every step of the way, including sign-up, payment, and support tickets. The email setups at HostByBelle and at YetiNode differ slightly. HostByBelle uses a transactional email provider, and YetiNode self-hosts its email MTA.

• Payment Providers

FOSSBilling at HostByBelle offered Paypal and payment from account credit. Payment wasn't involved with YetiNode, because I tested their free tier instead of their paid tier. Paypal payment worked fine at HostByBelle.

• Provisioning Shared Hosting and VPSes

Like most billing panels, FOSSBilling requires the use of another additional software package to provision hosting services. Sometimes, modules can be added to billing panels to send instructions to the additional provisioning software's API.

As shown at both HostByBelle and at YetiNode, FOSSBilling works well with HestiaCP for shared hosting. FOSSBilling's five (including Hestia) already-working provisioning integrations are listed in the FOSSBilling Documentation. Integration of Proxmox also has been discussed on the FOSSBilling Forum and, more recently (8/21/2023), on the FOSSBilling Discord.

• FOSSBilling Forum

If you have additional questions about FOSSBilling, you might want to check out the FOSSBilling Forum.

HestiaCP

• App Installation

As noted above, both HostByBelle and YetiNode use Hestia Control Panel for shared hosting installs. Here is HestiaCP's One Click App Installation Panel as seen at YetiNode.com. As expected, the HestiaCP app installation is similar at HostByBelle.

• Let's Encrypt Certificates

SSL certificates from Let's Encrypt were provisioned with one click at both HostByBelle and YetiNode.

FOSSBilling Updates

Both FOSSBilling's client theme and its multiple administrative user permissions hopefully will be updated in the next 0.6.0 release or soon thereafter. Belle posted screenshots of the updated client theme on FOSSBilling's Discord.

Hints From Belle

Belle said to me, about FOSSBilling, "If someone is going to use it, they should be technical and ideally be able to handle some PHP development."

Speaking of updates and changes, Belle advises FOSSBilling users to review the FOSSBilling changelog before updating. Changes occasionally affect people with customized installations, but people "using FOSSBilling as-is out of the box shouldn't really have such issues."

Last, but not least, Belle also advises reading the FOSSBilling FAQ.

Conclusion

FOSSBilling quickly and efficiently handled sign up, ordering, invoicing, and passing me to HestiaCP for provisioning. Support tickets and transactional emails from HostByBelle all worked as expected. FOSSBilling successfully kept track of payment and support transactions and also showed the history of each in clear interfaces. With HestiaCP, I quickly installed my above linked html test pages at HostByBelle and at YetiNode. Security certificates, like everything else, were easy.

Belle was super friendly and helpful via the HostByBelle FOSSBilling Support Tickets. @AuroraZero was, as always, wonderful, both via LES PMs and via email. Both HostByBelle and YetiNode are recommended for shared hosting.

FOSSBilling worked excellently from the perspective of a tester of shared hosting. The new era of a working, free, and open source billing platform has begun!

"Exited and prout to be posing my fist ever particle on LES."

@AlwaysSkint

Table Of Contents (T.O.C.):

1. Introduction and a bit of history
2. Hard Disk working principle
   2.1. CMR vs SMR recording technology
3. The physical size
4. Connection interface standards
   4.1. ATA
   4.2. SCSI and SAS
   4.3. SATA
   4.4. NVMe
   4.5. External drive connection standards
5. Hard disk performance basics
6. Hard disk technical specs
   6.1. Cache memory size
   6.2. Platter rotation speed
   6.3. Seek time and latency
   6.4. Sequential and random data transfer speed
   6.5. Durability stats
   6.6. Physical characteristics
   6.7. Capacity
   6.8. Air vs Helium
7. Hard disk buying recommendations
8. Conclusion

1. Introduction and a bit of history

The 10-second attention span people who like to "just skim" articles might want to skip straight to the chapter 2.

Relja BrevityIsNotMyVirtue Novović

There was a time when people used tapes to store large amounts of data. The main downside of using a tape for storage is the difficulty of finding a certain file, without reading the entire tape - (re)winding only gets you so far. It was much like playing a music cassette tape or a gramophone record.

In 1957, IBM built the first commercially available hard disks. HDDs allowed people to read, edit, delete, or add data to any file stored on the drive "right away." This is called "random access" and we take it for granted nowadays, but it was a major breakthrough in storage technology.

Over the following two decades, tape storage fought hard, but eventually lost in the vast majority of categories. There still are use cases where tape storage is the optimal choice, but that is a topic for a separate article.

Of course, the first hard disks were behemoths, used in enterprise server rooms and resembling modern day washing machines more than anything else. Their 24 inch large media was spun using powerful motors. Fun fact: my father and I put one such hard disk motor on our lawn mower, back in the 90s when there was a fuel shortage in my country. That thing is still running nicely at the time of writing, using three-phase 50 Hz 220V power.

An old hard disk motor - 1 horsepower, 3-phase power supply.
...For the life of me, I can’t remember if it was from a Sperry-Univac, or a decade younger Honeywell-Bull system...
Picture 1

Compare that to today’s 3.5" hard drives (and their smaller 2.5" laptop-friendly cousins). We’ve certainly come a long way. Nowadays, Solid State Drives (SSD) have replaced the traditional hard drive as the norm for most desktop computers and laptops. However, for storing large amounts of data, the good old hard disks are pretty much alive and kicking - with some more tech. improvements over the past few years. They are the stars of this article.

- T.O.C. -

2. Hard Disk working principle

I’ve seen programmers who don’t understand these concepts make an application that saves thousands of several-byte-small log files (i.e. very small files), and then get surprised to see the application take a huge amount of storage space. So, despite countless super-fast online programming courses, it helps to put some effort into understanding the hardware your stuff will be running on.

Relja Novović - shower thoughts

Let me briefly touch upon how these things work. A minute spent on the following text and pics should help you more easily understand hard disk technical specs. Ready?

Hard disks, as their name says, have discs covered in magnetic material which can be magnetised to north (N) or south (S) polarity. These discs are also called platters. There can be two or more platters in a hard disk drive (modern 3.5" drives can have up to five platters).

Similarly to a gramophone, the platters spin, and there are heads that can move across their surface to perform reading or writing.
Thanks to the damn hipsters, I hope that even the younger readers know what a gramophone is…

Hard disk drive's basic parts drawing
Picture 2

In the picture above, the DISK spins at over 5,000 rpm, while the R/W HEAD moves left-right, floating just about 10nm above its surface (and reading or writing data to it by using electromagnetic induction). When a hard disk drive is turned off, its heads are locked in place so they don’t hit the platters if the drive is transported. This is called parking the heads.

This part serves to introduce you to the technical terms you may hear: cylinders, tracks, and sectors. Alas, when you format a hard disk drive, you are logically dividing each platter into concentric tracks (imagine race tracks at the Olympics), and dividing each track into sectors (smaller sections along the track).

Every sector contains a set number of bytes (groups of eight digits that can only be a logical one or a logical zero). It’s usually 512 bytes on most file systems. Several sectors are grouped into a cluster, the smallest amount of disk’s storage space that can be taken by one file. If your cluster is, say 4096 bytes size, and you save a file that is only 1024 bytes large, it will still take up the entire cluster. However, if a file is 8192 bytes large, it will span across two 4094-byte clusters.

Before I totally confuse you by explaining what a cylinder is, a picture will explain it better:

Hard disk's logical division: cylinders, tracks and sectors
Image source: Wikipedia
Picture 3

So, now you clearly understand what a cylinder is: a set of tracks on the same distance from the platter’s centre, across all the platters in the drive. Cylinders are no longer used for addressing hard disk storage - they have been replaced with logical block addressing (LBA), but that's a topic for a separate article.

Here's what that looks like in real life:

Hard disk internals photo
Picture 4

Well, I hope you are now smarter than you were this morning. Let us move on to the more “flashy” stuff.

- T.O.C. -

2.1. CMR vs SMR recording technology

This one is rather important, especially if you value performance and reliability.

Remember what we’ve just said about tracks and how those are created in concentric circles on the platters when you format a hard drive? Each track contains sectors where bits are written and read from. That’s the old style, called CMR (Conventional Magnetic Recording).

Fact: you need more room (more track width) to write a bit to a track (i.e. polarize its magnetisation) than you need to read from it. That is why some genius thought it was a jolly good idea to have the tracks overlap! That is how SMR (Shingled Magnetic Recording) drives were born.

CMR vs SMR system of writing and reading data from a hard disk's platter
Picture 5

SMR lets you write about 25% more data on a given platter size, which lets you build larger capacity drives at a lower cost. What could possibly go wrong?

Well, SMR relies on the fact that the head for reading can work with a narrower track. But when you need to write to a track, you will also rewrite the adjacent tracks. So, if you write data on an SMR drive track, and its adjacent tracks have some data, you would need to first read and rewrite that data too, in order to prevent data corruption. This makes SMR drives perform awfully poorly when writing if they are loaded with data.

Some manufacturers don't disclose when their drives are built using the SMR system. Make sure to double check and avoid buying such drives - at least that's what I do and recommend.

- T.O.C. -

3. The physical size

3.5 inches is quite large.

Japanese proverb

Today, hard disks come in two formats: 2.5" and 3.5".

2.5" are the smaller ones with platters that spin slower, make less noise, heat and use less power. They are designed for use in laptops and portable (external) hard drive cases, and a part of this design is a more rugged mechanism of parking drive heads to avoid any damage in case of bumps or similar. Thanks to their lower power consumption, they can use a USB cable for power supply (handy for portable drives). An exception are server-grade 2.5" drives built to be compact, but work 24/7 in servers. Either option is more expensive per TB of storage space compared to 3.5" drives.

3.5" is the usual size for drives designed to be placed in desktop computers and many servers and NAS drives. These are generally faster, louder, require more power, but offer higher storage capacities for the price.

- T.O.C. -

4. Connection interface standards

The nicest thing about standards is that there are so many of them to choose from.

Ken Olsen

Gotta plug it in somehow, right? Let us go over some of the most widely used standards, starting with a couple of obsolete ones. Note that the given data transfer speed info is related to the connection's capacity, it doesn't mean every (or any compatible) drive can reach that speed.

- T.O.C. -

4.1. ATA

An obsolete standard. ATA (Advanced Technology Attachment) is the standardized name for what was also called PATA (Parallel ATA) and IDE (Integrated Drive Electronics). It uses a parallel connection, meaning several bits are transferred all at once. That is why ATA cables are flat and very wide.

ATA data transfer speeds go up to 133 MB/s.

- T.O.C. -

4.2. SCSI and SAS

SCSI (pronounced as “scuzzy”) stands for “Small Computer System Interface.” The standard started gaining popularity during the ‘80s (20th century) as a fast, high-quality standard for enterprise storage, because it allowed a connection of a relatively large number of drives, often placed in RAID (Redundant Array of Inexpensive Disks).

It started as a high-speed parallel connection, but got updated over the years. Its latest version uses a serial connection. It’s called “Modern Serial Attached SCSI” (SAS) and the latest models reach data transfer speeds of up to 1,875 MB/s. This stuff is (and always has been) expensive, and can be found in servers.

SAS interface can also accept SATA drives (though SATA interface won’t accept any SAS drives).

- T.O.C. -

4.3. SATA

SATA (Serial Advanced Technology Attachment). This is the de facto current desktop and laptop storage connection standard. It uses a serial connection at a high frequency (and speed), and allows for thin and long (up to one metre) cables.

SATA data transfer speeds go up to 600 MB/s (the latest, SATA III standard).

This interface is fast enough for most consumer-grade hard drives. If you are buying a HDD today, and reading this, you are most likely looking for a SATA HDD.

- T.O.C. -

4.4. NVMe

NVMe (Non-volatile Memory Express) was originally designed to work with SSDs (Solid State Drives). Unlike other storage media interfaces, NVMe lets you connect your drives directly to the CPU, using the PCIe interface. This allows for very low latencies and huge data transfer speeds of over 10,000 MB/s over PCIe 5.0.

At the time of writing, it is mostly used for SSDs (using M.2 connection format), though some companies, like Seagate, are working on building a HDD that natively connects via NVMe.

If you’d like to know more, Western Digital’s blog has a great article on NVMe.

- T.O.C. -

4.5. External drive connection standards

Let’s do a quick overview of the connection standards for external drives (portable drives, NAS expansion units, and similar).

FireWire
A standard launched by Apple, which has now become obsolete for reasons beyond the scope of this article. FireWire 800 standard could reach data transfer speeds of up to 100 MB/s.

USB
Current standard for connecting many external 2.5” drives is USB 3.1 with data transfer speeds of up to 1,250 MB/s, while the newer USB 3.2 can go up to 2,500 MB/s (older USB 1.0 and 2.0 standards are now practically obsolete).

Even newer USB4 standard comes in 20 Gbps (that is giga-bits, not giga-bytes per second) and 40 Gbps (5,000 MB/s - that is mega-bytes per second) variants (the 40 Gbps version requires the more expensive 40 Gbps cables).

There should also be a USB4 v 2.0 standard with speeds of 80 Gbps (10,000 MB/s), but I have never seen any appliances or even cables for that one.

Note, there is a USB C connector (physical connector) standard that often gets confused with the transfer speed standards of USB 3.1, 3.2, USB4 etc, but you technically could have the older, slower USB 3.1 working over a USB C physical connector.

USB is capable of supplying power over the same cable. USB A connector cables can supply up to 7.5W, while USB C connector cables can supply up to 100W or even 240W with the latest USB-C PD connector standard.

Thunderbolt
This standard requires that your motherboard supports it (which is not as commonly the case as it is with USB 3.2 nowadays). It offers speeds of up to 40 Gbit/s (5,000 MB/s).

It connects by combining the PCIe and DP (DisplayPort) signals into two serial signals, and also provides the needed power supply of up to 100W - all using the same cable.

In practice, Thunderbolt connectors and equipment are not very widely used, at least in my experience.

eSATA
Another standard that is getting obsolete (though Synology uses it for connecting some of their NAS expansion units). It doesn’t supply any power (so you need a separate power supply for the eSATA drives), though there is the eSATAp standard which does.

In its latest & greatest version, eSATA standard provides data transfer speeds of up to 600 MB/s.

I think this covers the standards you are most likely to encounter. Let us move on.

Connector types overview
Picture 6

- T.O.C. -

5. Hard disk performance basics

A child of five would understand this. Send someone to fetch a child of five.

Groucho Marx

Hard disks use spinning platters to store data. A read/write head moves up-down over the platter surface (see picture 2 for details). That is why they can read or write a lot faster if they are doing it all in one go. That is called sequential access reading/writing. If, on the other hand, they need to read a file (or files) dispersed over several non-adjacent sectors, that kind of reading/writing, called random access, is much slower.

Likewise, because reading only requires to, well, read the magnetic polarity of bits, while writing requires actual magnetisation of each bit, hard disk read speeds are a lot faster than their write speeds.

So, even if you buy the latest, greatest and fastest hard disk drive in the world, you should expect its random access speeds to be lower than its sequential access speeds, and its writing speeds to be lower than its reading speeds.

Manufacturers often state the most “optimistic” scenarios in their selling brochures, so it’s good to keep this in mind and double-check.

- T.O.C. -

6. Hard disk technical specs

There is no amount of memory that human stupidity can’t fill up.

Živojin Novović, when we discussed our home PC’s hard disk upgrade

In this “chapter,” I will cover the basic hard disk drive technical specifications, explain what each one means, how it affects the overall performance, and what its ballpark values are. I’ll use SATA drives as examples, since those are the most popular consumer and “prosumer” drives at the time of writing.

For brevity, I will not discuss every existing aspect of hard drives, but concentrate on the stuff you can quickly check and compare when deciding which drive to purchase.

6.1. Cache memory size

I have already explained what caching is (the context of that article was about website page caching, but the principle is the same). It boils down to using fast storage, such as RAM (Random Access Memory), to quickly store and retrieve data.

For a hard disk, this means: when you want to save a file, you just send it to the drive’s cache and call it a day, letting the drive worry about writing it to its platters for a permanent storage.

A similar thing happens when you are reading data from the drive. Say you open the document “pigeon-deliveries-june.doc”. Drives have smart controllers that can predict the likelihood of you requesting a file such as "pigeon-deliveries-july.doc" next, letting them take the time to read that file and load it into the drives cache, so you can retrieve it quickly if you decide to.

Caching, especially for reading files, is a hit-and-miss, but it can often help provide a faster perceived performance - i.e. despite the drive having the same read speed, you will effectively read and write your files faster, thanks to the use of cache.

Of course, cache memory is a lot more expensive than platter storage space, so even 1,048,576MB (about 1TB) large drives often come with only 64 MB of cache. That is why cache can’t help when reading or writing large amounts of data. Having said that, since hard disks are the slowest when writing, and most users write (save) a lot of smaller files at irregular intervals, cache helps a lot in practice.

Cache size usually depends on the drive's total storage capacity (larger drives have more cache memory). Typical cache size ranges from 32MB for lower end 1TB drives, to 256MB for larger capacity or high-end drives. More cache costs more, but manufacturers usually provide the optimal ratio based on drive’s capacity and intended use.

- T.O.C. -

6.2. Platter rotation speed

The faster drive’s platters spin, the faster you can read/write data (sequential read/write), and the less time it takes for the sector with the needed data to reach the reading/writing head (for random access). However, higher rotation speed also requires more power, and creates more heat and noise.

Typical speeds are 5,400 and 7,200 RPM, though some enterprise (SAS) drives spin up to 15,000 RPM.

Because of the extra noise, faster needn’t always be better, it depends on your use case.

- T.O.C. -

6.3. Seek time and latency

We know that read/write heads move across the platters. This takes time. Moving from one track to the next is a lot faster than moving from the outermost all the way to the innermost track. So, we have:

• Track-to-track seek time: often below 2ms.
• Average seek time: often about 5ms.

For some reason, seek time specs are next to impossible to get for most modern drives.

Now, since the platters are spinning, it might take some time for the sector you need to come under the reading head. Average time taken for this to happen is called the average rotational latency. For a drive that spins at 5,400 RPM, that is 60 (seconds) divided by 5,400, divided by 2 (to get the average). So, about 5.55ms. This goes down to 4.16ms for 7,200 RPM drives, and as low as 2ms for the 15,000 RPM drives.

Do not mix this rotational latency with the average read/write latency, which is:

• how long it takes for the drive's controller to figure out where the required block of data is on the platters,
• move the read/write head to it,
• and start data read/write process (transfer).

Average read latencies often go to about 30ms (and max. latencies can be as high as up to one second), while average write latencies can be ten times longer, so about 300ms.

These stats affect random read/write performance, but don’t affect sequential read/write performance.

- T.O.C. -

6.4. Sequential and random data transfer speed

Data transfer speed provided by manufacturers is basically average sequential read/write speed. It is often around 200 MB/s for modern drives. Drives that spin faster usually have a bit higher "data transfer speed" (i.e. average sequential read/write data transfer rate).

Random read/write performance gives data transfer speeds that are about 1% (one percent) of the sequential read/write speeds (so about 2 MB/s). These are often expressed in IOPS (Inputs and Outputs Per Second). In another article I've explained IOPS in more detail.

IOPS results differ based on the size of data blocks being read or written, but let's say that ballpark values are in hundreds of IOPS, and that with modern drives, write performance gives more IOPS thanks to the good use of caching. When a disk reads, it needs to read the given files wherever they are, often doing a lot of back-and-forth. When it writes, the data is stored in cache and then written in a more optimized way (less "random" write head movement).

- T.O.C. -

6.5. Durability stats

These are the stats related to durability. Note that these stats have more to do with the drive’s intended use than its actual quality (I’ll go to recommend models later on).

Power on hours per year
8,760 is for drives designed to work 24/7 (NAS and enterprise/server drives).
2,400 is for drives intended for personal or business computers (about 8 hours of daily use).

Unrecoverable read errors per bits read (URE)
This is basically the probability of a drive not being able to read a piece of written data (even with error correction attempted). 1 bit errror in 10^14 bits read is the usual number for consumer grade drives, while 1 in 10^15 or 10^16 is more common for NAS and enterprise/server grade drives.

In theory, 1 in 10^14 means there will be an error for every ~ 12 TB of data written, but as I said at the start of this “chapter,” this info has more to do with a drive’s intended use and manufacturer’s warranty, than it does with the actual durability and reliability of the drive.

Workload Rate Limit (WRL)
How many terabytes (TB) per year you can read or write to the drive. Consumer grade stuff is good for about 50TB, while the enterprise stuff is rated at 150TB per year or more.

Mean Time Between Failures (MTBF)
The arithmetic mean (average) of time between failures. Usually expressed in hours, and ranges around one million. Based on my knowledge and experience, this metric can be disregarded.

A stat that is actually useful (but it is not provided by drive manufacturers) is the Average Failure Rate (AFR). Unfortunately, to get this data, you would need a ton of input from service centres around the world. Or rely on Backblaze quarterly stats (Backblaze 2023 Q2 stats link) if they have used a drive that you are looking to buy.

- T.O.C. -

6.6. Physical characteristics

Apart from the size (form factor) explained in chapter 3, these are some other important disk drive physical characteristics.

Power consumption
How many watts the drive consumes when working on average (from about 5W to 10W or more for the larger enterprise grade drives). Faster spinning drives consume more power. Air filled drives (as opposed to helium filled ones) consume more power, since air creates more drag.

Noise (Acoustics)
Drives make noise when they are spinning, and even more noise when they are reading or writing. This noise level is expressed in A-weighted decibels (dBA). dB scale is logarithmic, meaning a 10 dB increase means 10 times louder, while a 20 dB increase means 100 times louder (10 times 10).

For example, 20 dBA is whisper, 60 dBA is normal conversation, and 100 dBA is a loud motorcycle.

Most drives are below 30 dBA when idling, and above 30 dBA when reading. But even a couple of dBA difference is noticeable, as explained in the previous paragraph. 20 dBA idle and 25 dBA when reading/writing is considered to be very quiet for hard drives.

- T.O.C. -

6.7. Capacity

Modern hard disk storage capacity is expressed in terabytes (TB). TB is a thousand megabytes (MB). Some smaller models have sizes of 500 MB or similar, but those are getting obsolete. Modern drives range from one to a dozen or more TB.

Larger capacity drives often have more platters, consume more power and cost more.

- T.O.C. -

6.8. Air vs Helium

For decades, hard disks had a small hole (with a dust filter) to let the air move in and out (as the inside air temperature changes). Read/write heads float at about a dozen nanometres above the platters. If the air becomes too thin (at high altitudes), air-filled drives will not work properly, their heads will drop and scratch the platters.

Helium, unlike air, is more stable and provides a lot less drag as the drive platters spin (and its read/write head moves).

The problem with helium is that its molecules are so tiny, that it is very difficult to keep it from seeping out of the hard drive’s case. Some extraordinary high tech. advancements over the past decade have allowed manufacturers to seal their hard disk drive units well enough for the damn gas to stay inside. And voila, today we have helium filled drives.

Helium filled drives make less noise, use less power (all thanks to the lower drag), and, because they are completely sealed, can operate safely in low pressure environments. Making the tight seal costs more, so at the time of writing, only high-capacity drives (10TB or more) come helium filled.

- T.O.C. -

7. Hard disk buying recommendations

I am not rich enough to buy cheap stuff.

English proverb ( if this one doesn't get me banned, nothing will! )

In this article I'll say what I would buy (or have bought). I am not in a position to test dozens of models, but I work with computers for a living, and I like listening to people: service technicians, system administrators (and the good folks of the LowEndSpirit forum ).

How big?
If at all possible, for sizes of 2TB or smaller, do not buy a hard disk drive, get an NVMe or at least a SATA SSD. Today, hard disks make sense for 4TB or larger sizes.

For a NVMe SSD, I use and recommend Samsung EVO 970 Plus (Amazon affiliate link). It costs well under $100 for the 2TB version, it is super-reliable and far from slow, even under high loads. Note that it supports PCIe 3 speeds only, but for most use cases that means less heat, and performance that is more than good enough.

PCIe 4 gives a sequential read/write speed boost due to a higher bandwidth, but for general use (i.e. many random read/write operations) it gives next to no benefits (also, PCIe 3 NVMe drives are also pretty fast even for sequential read/write).

My opinion:
At the time of writing I can not recommend a PCIe 4 (or PCIe 5) NVMe drives in good faith. Many models of many manufacturers have problems with heating and reliability - and as the heat rises, you throttle back to PCIe 3 speeds or lower. Samsung has a price premium which is more than worth it for the above-recommended EVO 970 Plus model, but their PCIe4 drives had problems and I'm still not sure how reliable they are despite the firmware updates published after a public outrage.
UPDATE, December 2023:
Samsung 990 PRO Heatsink (Amazon affiliate link) looks promising (I’m yet to test it personally). Hardware-supported encryption, and, as far as I know, no firmware-related problems that plagued some Samsung NVMe models. It comes in a 4TB version (along with the 1TB and 2TB alternatives), and a built-in heatsink (good cooling is important for PCIe Gen4 speeds). If I were buying a new system drive today, I’d go with the 4TB version of this one.

Which brand (manufacturer)?
I'll start with Toshiba. Their drives are on the noisier side, but I've had great results with them in terms of reliability and durability. Batches that are sold in my country (and Hungary for that matter) seem to have the lowest percentage of "lemons."

Seagate is another decent manufacturer I've had good results with.

Western Digital (WD) is a highly renowned manufacturer, but I have not had very good results with them.

The thing is: I am only one person. Sure, decades of experience, but that still doesn't make for a statistically useful data. Also, many manufacturers make a blunder every now and then. So I would argue that the manufacturer is not nearly as important as the particular model. At the time of buying, see which models perform well and are reliable. If you can normally go and talk to people who deal with warranties of large hardware stores, they will probably be able to tell you which models get the lowest number of returns (though, keep in mind that expensive models get sold a lot less, so them having fewer returns doesn't always mean they are more reliable).

Concrete model recommendations
See the paragraph above to understand the limitations of recommendations like this. With that out of the way, you should also know that most manufacturers have turned to SMR recording technology for their consumer-grade drives. I would recommend you avoid SMR drives. They are not worth the savings.

My opinion:
That makes perfect sense in capitalism: making the low and mid-priced stuff be less durable and less reliable, and practically "forcing" people to buy the more expensive stuff.

For larger storage capacities, I would recommend Seagate IronWolf 12TB NAS (Amazon affiliate link - 'cause yachts won't pay for themselves! - though for the US folks, I would recommend buying directly from the manufacturer to get a better and longer warranty protection) or higher capacity IronWolf models if you need more storage.

If you wish to save some money or don't need that much storage space, you could go with a smaller, Seagate IronWolf 4TB NAS drive (Amazon affiliate link).

Alternatively, if you don't mind a bit more noise, you can't go wrong with Toshiba N-series drives, like Toshiba N300 4TB NAS (Amazon affiliate link).

- T.O.C. -

8. Conclusion

As with many of my other articles, this one too serves primarily as my own reminder and reference. I will try to keep it up-to-date and correct any errors - my articles aren't written by AI or "content writers," and English is not my native as you have probably figured out by now.

If you have questions, additions and, especially, corrections - please post a comment.

- T.O.C. -

Relja Novović - about the author:
https://www.bikegremlin.com/about/

What?

Plan 9 from Bell Labs is an operating system from the creators of the C Programming Language and the Unix operating system.

Although Plan 9 revolutionized many aspects of Unix, it nevertheless still remains possible in 2025 for Unix fans to access aspects of old Unix by installing Plan 9 inside Qemu and changing the Plan 9 boot procedure so that Plan 9 stays in the terminal rather than starting a graphical monitor.

Why?

• This method provides a text-only terminal with "infinite" scrollback. Infinite scrollback is how paper terminals used to work before the curses interface and the Graphical User Interface ("GUI") were invented.

• Plan 9 runs on the i386 processor architecture in addition to many other architectures.

• Plan 9 has very few system calls compared to other operating systems.

• Plan 9 comes with full source code and build tools.

• The code for Unix userland utilities which remain in Plan 9 is shorter than later versions.

• Plan 9 is fast!

Why not?

• This method, which perhaps involves the conflation of text and terminal, is arguably a complete misunderstanding of everything that's truly the greatest about Plan 9. See this excellent discussion (but be a little careful when reading, because possibly not everything that's said is 100% correct).

How?

Download Plan 9. Options include the classic Plan 9 from Bell Labs (used here, see links just below), as well as, probably, 9legacy and 9front, neither of which were tested for this post.

wget http://9p.io/plan9checksums.txt 
wget http://9p.io/plan9/download/plan9.iso.bz2

# md5 sum for the bz2 file used here:
# 5183b2d0f11fded871461416f8c78cfb  plan9.iso.bz2

Make a qemu disk image for our Plan 9 install.

# qemu-img create -f qcow2 plan9-console.qcow2 10G

Use your favorite VNC app set to localhost:5900. Relaunch the app if it crashes when Plan 9 switches from terminal to grraphics.

# ssh tom@toto -L 5900:localhost:5900 # VNC via ssh tunnel to your server from your local machine.

Install Plan 9. Usually it's easy, just boot the ISO and accept the defaults. There are lots of tutorials and videos about the installing Plan 9. Here below is a qemu command to start the install. This command doesn't use KVM. Not using KVM makes slower output, which is easier to read, but you might need to wait a minute for longer operations. If you wish, add KVM with -enable-kvm.

qemu-system-i386 \ 
  -vnc 127.0.0.1:0 \ 
  -hda plan9-console.qcow2 \
  -cdrom plan9.iso \
  -boot d

After Plan 9 is installed, you see the graphical Plan 9 (screenshot above). Edit plan9.ini with the Plan 9 VT-100 emulator and the ed editor.

# In the Plan 9 rio terminal, type vt
# In the vt terminal, type '9fat:' (Don't forget the colon!)
# ed /n/9fat/plan9.ini : 
#   - remove last two lines (monitor, and size)
#   - add a line which says console=0

After you change plan9.ini, you don't need VNC, just ssh into your server and run Qemu.

qemu-system-i386 \ # Not using KVM makes slower output, which is easier to read.
  -nographic \
  -hda plan9-console.qcow2 \
  -boot c

Here below is the output of the above. We can launch the Plan 9 versions of old Unix commands like awk and ed. We can get infinite scrollback, almost like a paper terminal.

tom@toto:~/plan9$ cat 20250228a-boot-messages
# 20250228 Boot Plan9 in Qemu with plan9.ini changed to remain in host node shell

tom@toto:~/plan9$ ./start-plan9-console

SeaBIOS (version 1.16.3-debian-1.16.3-2)


iPXE (https://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+06FC6F60+06F06F60 CA00



Booting from Hard Disk...
MBR...PBS1...
Plan 9 from Bell Labpcirouting: BIOS workaround: PCI.0.1.3 at pin 1 link 96 irq 10 -> 9
 disk loader

cpu0:  3091MHz GenuineIntel Celeron (cpuid: AX 0x0663 DX 0x781ABFD)
ELCR: 0C00
497M memory: 497M kernel data, 0M user, 18M swap
found partition #S/sdC0/data 0 20,971,520
disks: sdC0 sdD0
trying sdC0....found 9pcf
.1121326.....................................................................................................0
entry: 0xf0100020

Plan 9
E820: 00000000 0009fc00 memory
E820: 0009fc00 000a0000 reserved
E820: 000f0000 00100000 reserved
E820: 00100000 07fe0000 memory
E820: 07fe0000 08000000 reserved
E820: fffc0000 100000000 reserved
cpu0:  3096MHz GenuineIntel Celeron (cpuid: AX 0x0663 DX 0x781ABFD)
ELCR: 0C00
igbe: p->cls 0x0, setting to 0x10
#l0: i82543: 1Gbps port 0xFEB00000 irq 11: 525400123456
128M memory: 54M kernel data, 74M user, 297M swap
usbinit...usbd.../boot/usbd: /dev/usb: no hubs
no /srv/usb...no usb disk...pickmethod...read #e/nobootprompt...read #e/bootargs...outin...root is from (tcp,[ . . . copy paste error] 
pickmethod done
bind #æ...bind #S...partinit...auth...user[none]: glenda
usbinit...usbd.../boot/usbd: /dev/usb: no hubs
no /srv/usb...no usb disk...mount usbd...boot: can't open /srv/usb: '/srv/usb' file does not exist
time...
fossil(#S/sdC0/fossil)...version...can't stat /srv/partfs.sdXX: '/srv/partfs.sdXX' file does not exist

init: starting /bin/rc
rio: can't open display: initdisplay: /dev/draw/new: no frame buffer
init: rc exit status: rio 30: display open

init: starting /bin/rc
term% who am i
glenda
term% awk 
Usage: awk [-F fieldsep] [-mf n] [-mr n] [-v var=value] [-f programfile | 'program'] [file ...]
term% ed
a
hello
.
w hello
6
Q
term% ls -l hello
--rw-rw-r-- M 8 glenda glenda 6 Feb 28 21:07 hello
term% fshalt
syncing.../srv/fscons...prompt: 
halting.../srv/fscons...
prompt: 
done halting
Ctrl-a x # Quit qemu
tom@toto:~/plan9$ 

Linveo kindly provided several free VPSes to guys actively posting in the LES BSD Thread. Linveo also kindly added @cmeerw's NetBSD 10 Minimum image. As shown above, NetBSD 10 Minimum now easily can be installed at Linveo directly from "Other BSD" category on the Rebuild page of vmcontrol.linveo.com.

After installing @cmeerw's NetBSD 10 Minimum image, with help from the NetBSD Guide references linked below, I followed the steps listed below to upgrade from NetBSD 10 to self-compiled NetBSD-current.

Getting to self-compiled NetBSD-current requires us to add to the NetBSD 10 Minimum VPS a NetBSD 10 compiler, other build tools, and the source code for both the NetBSD-current kernel and the NetBSD-current userland. Then a NetBSD-current compiler and other build tools are compiled by the NetBSD 10 compiler. Next, the newly compiled NetBSD-current build tools are used to build the NetBSD-current kernel and the NetBSD-current userland. The end result is a NetBSD-current VPS running its own code as compiled on itself by its own compiler.

My Linveo VPS seems to reboot and to run normally following the upgrade to NetBSD-current. There seems to be plenty of space remaining available, though the X sources and pkgsrc haven't yet been added:

linveo# df -h .
Filesystem     Size   Used  Avail %Cap Mounted on
/dev/dk2        48G    10G    35G  22% /
linveo# 

linveo$ CVS_RSH=ssh cvs -danoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P pkgsrc
  [ . . . ]
linveo$ df -h .
Filesystem     Size   Used  Avail %Cap Mounted on
/dev/dk2        48G    11G    35G  24% /
linveo$ 

• Issue

The biggest issue that I know about with respect to the NetBSD-current upgrade proposed here relates to the etcupdate command, which is the last command in the sequence below.

etcupdate is an sh shell script. There also is a man page, which mentions that the idea for etcupdate came from FreeBSD's mergemster.

I need to figure out how to set options making etcupdate faster and easier to run within the limited scope of this single CPU architecture NetBSD-current upgrade. The VPS is not attempting to build the full multi-architecture NetBSD panopoly. Perhaps limiting the etcupdate source directory to my VM's amd64 architecture might help. Perhaps -- as recommended in the man page -- running etcupdate on a tar file instead of directly on the sources also might help.

• NetBSD 10 Image

Here are links to @cmeerw's NetBSD 10 image and to a discussion of how the image was made:

https://lowendspirit.com/discussion/comment/187386/#Comment_187386
https://lowendspirit.com/discussion/comment/187452/#Comment_187452

• NetBSD Guide Reference Links

Here are reference links to the very helpful NetBSD Guide:

https://www.netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-cvs-netbsd-release
https://www.netbsd.org/docs/guide/en/chap-updating.html

• Adding Compilers To The NetBSD 10 Image

Here is the LES comment where @cmeerw showed us the one line command used below to add NetBSD 10 compilers to the NetBSD 10 base image:

https://lowendspirit.com/discussion/comment/186774/#Comment_186774

• Recipe

Here are the steps I followed to convert a clean, new Linveo VPS install of @cmeerw's NetBSD 10 image to what seems to be self-compiled NetBSD-current:

• Run as root to install a compiler, create needed directories, and an unprivileged user

ftp -o - https://cdn.netbsd.org/pub/NetBSD/NetBSD-10.0/amd64/binary/sets/comp.tar.xz | progress -z tar xpf - -C /
cd /usr
mkdir obj src tools
mkdir /home
useradd -m -G wheel tom
passwd tom
chown tom obj src tools
su tom 

• Run as the unprivileged user which owns the obj, src, and tools subdirectories

export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
export CVS_RSH="ssh"
cvs checkout -A -P src
cd src
less UPDATING
./build.sh -j 2 -O ../obj -T ../tools -U distribution
./build.sh -j 2 -O ../obj -T ../tools -U kernel=GENERIC

• Back to root to install and reboot to use our newly compiled NetBSD-current kernel

mv /netbsd /netbsd.old
mv /usr/obj/sys/arch/amd64/compile/GENERIC/netbsd /
shutdown -r now

• As root, install and reboot to run the newly compiled NetBSD-current userland

cd /usr/src
./build.sh -j 2 -O ../obj -T ../tools -U install=/
shutdown -r now

• As root, postinstall

/usr/sbin/postinstall -s /usr/src check
/usr/sbin/postinstall -s /usr/src fix

• As root, etcupdate

/usr/sbin/etcupdate -s /usr/src

Please point out any mistakes.

Any questions, please ask.

Thanks @linveo for the nice VPS! Thanks @cmeerw for the NetBSD 10 Minimum image! Thanks to the NetBSD project! And thanks to LES!

Why

The gear I've got for firewall is stupidly overkill (optane drive & 16GB RAM) for something that is 99% idle so I want to stick a couple other things on the same hardware via proxmox.

Assumed knowledge and requirements

Proxmox, linux cli, basic networking, (very basic) firewall knowledge. Nothing here is super hardcore technically as such...just unforgiving in sequence & details if unfamiliar hence notes. i.e. when it doesn't work it's not obvious where you fkd up.

Bring backups & loads of time. Frankly...assume that you'll fu.ck this up if you haven't done this before. So this is a Saturday morning project not a Sunday night one.

Before you start set up a ventoy USB stick adding both proxmox and opnsense ISO on there (plus perhaps a PDF copy of this post). You may not have internet access while doing this so come prepared...

I'm using my own IP ranges etc in post - adjust as needed. Key pieces you need to know to follow my descriptions:

vmbr0 is LAN side
vmbr1 is WAN (internet) side. (Proxmox sticks VMs on 0 by default so you want LAN side being 0)
10.32.0.1 is the opnsense VM [LAN side], 10.31.0.5 is WAN side but that doesn't really matter key part is 31 makes it separate from our main 32 LAN
10.32.0.2 is the LAN side IP of proxmox
My device has 5 ports with enp2s0 being internet facing [aka vmbr1], rest of the NICs are bridge to vmbr0
The key networking file on proxmox is at /etc/network/interfaces

Assumes upstream ISP is Eth based and just DHCPs whatever is connected. If you've got something else on ISP side you're on your own (sorry)

Virtualized FW

There are two basic ways to do this. Either pass through the NICs to the FW VM, or do two sets of bridges with one being WAN (internet) side and one LAN side. Passthrough is tricky to get working on a basic linux/bios/hardware level but solves some security issues. I'm doing the bridge approach because my gear (oddly) doesn't seem to support pass through. This guide is bridge only...so abandon ship now if you're going passthrough.

Bridge has one significant security implication. Proxmox gets the external traffic and bridges it to the firewall VM. Grand. Except Proxmox is fond of serving the proxmox management interface on ALL interfaces. So it's serving the damn management GUI before anything gets to our FW. Meaning it is potentially exposed BEFORE our fancy opnsense security gets to it. That's a problem we're trying to get around with below section on messing with the Proxmox firewall...despite setting up a opnsense firewall. [In reality the WAN side has a pvt non-routable IP so should be fine even without rules but I don't 100% trust that]

Keep this in mind when doing say incoming wireguard...you're effectively double-firewalled...and would need to pass through wireguard on both.

Security

The bulk of this guide happens BEFORE anything is connected to the internet, mostly because above "I don't trust this" issue. Just the FW device and a laptop directly connected to it.

GO

Step 1: Install proxmox & ensure you can access the GUI

On the network section do:
IP 10.32.0.2
Gateway 10.32.0.1
Set DNS to 1.1.1.1
Select a NIC that is LAN side (enp7s0 for me) - this should be whatever your laptop is plug in to

The proxmox interface is on https://10.32.0.2:8006

Step 2: Set up a second bridge in /etc/network/interfaces with WAN

auto vmbr1
iface vmbr1 inet static
        address 10.31.0.5/24
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

iface enp2s0 inet manual

Note how it is set to a static unroutable IP despite us aiming for DHCP upstream ISP side. That's because we want the firewall VM to grab the DHCP from ISP, not the proxmox level interface.

Restart to make the 2nd bridge show in GUI.

Step 3: Set up a opnsense VM

Copy over ISO off USB stick like so:

mkdir /media/usbstick
fdisk -l  [[[to get the /dev/ location for the stick]]]
mount /dev/sda1 /media/usbstick
cp /media/usbstick/OPNSense.sio /var/lib/vz/template/iso/

If the iso is in that location then proxmox GUI should pick it up.

Set up a VM that has 12GB, 5120mb RAM and 3 cores. Going lower on space & ram works but has consequences downstream on how zfs plays out so if you can do 12 & 5 or more.

Before starting VM add the second bridge (vmbr1). Ideally add the LAN side during VM creation so that the naming 0 & 1 line up consistently. Memorize the MACs on both (sorta...just which is which).

Step 4: Install opnsense

Where it asks you for a login, use installer as user and opnsense as password

Key point here is you need to line up what opnsense thinks is LAN side and WAN side...with the right MAC per above...which in turn maps to the right bridge...which in turn maps to a physical port.

Finish install, change root pass & unmount the ISO. Reboot.

Step 5: Initial config of opnsense

Log into console of opnsense (the proxmox built in one). Option 2 - set interface IP. Configure LAN side. This part is a little confusing. We DONT want it to use DHCP as in where it gets it's IP, but we want it to SERVE DHCP...so that other LAN devices can get an IP from it. So in this menu the first time it asks you about DHCP the answer is no, the second time the answer is yes. Set IP to 10.32.0.1 and 24 as subnet bit count. Enable DHCP server. Start clients at 10.32.0.51 and end at 10.32.0.254. That way we've got ~50 IPs for statics. Change GUI to http. Reset to defaults - yes.

...now try to access GUI on http://10.32.0.1

On the GUI you should have an initial setup. Set hostname, domain, and DNS. Go with a public one for now e.g. 1.1.1.1 Leave the WAN side as DHCP - no need to change anything here.

Step 6: Firewall

Go back to proxmox & navigate to datacenter on left then to firewall. This stuff cascades so the rules you set on datacenter level apply to the device and to VMs below.

The firewall rules use first match principle, starting with zero i.e. top of list. So in general we want a pattern like so

[0] - Allow SSH on LAN side
[1] - Disallow SSH on everything

So lan side incoming SSH hits rule 0 matches that and gets approved, while everything else fails rule 0 then goes to rule 1 and gets dropped.

Direction: In
Action: Accept
Interface: vmbr0 [[[[this is LAN side]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick

Direction: In
Action: Drop
Interface: [[[[leave empty]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick

Do same for 22 TCP and ICMP (protocol, no port). I'd suggest messing with ICMP to experiment and get a better grasp of what blocks when.

Triple check that you've got this right. Next step is enable firewall...and if you fkd this (8006 specifically) up then you're locked out and will need a physically screen & keyboard to sort it out.

You should also block these ports on vmbr1 (internet/wan) for good measure:
TCP 111, UDP 111, UDP 323, TCP 25, TCP 3128. No idea what they all do...but proxmox is listening on them & I don't want them open internet facing. I used command "lsof -i -P -n" to work out what ports are being listened to.

Step 7: Connect device to internet

If all went well then a device plugged into LAN side can now access the internet. First thing you want to do is head over to

https://www.yougetsignal.com/tools/open-ports/

And confirm they can't see your port 8006

Trouble shooting

1) You'll need to set a static IP in the right range on your laptop when connecting directly to a device. So when you can't reach the interfaces (proxmox or opnsense) checking that you're on the right net is helpful. Also windows is really shtty on this so sometimes you need to disable and reenable the network adapter to make it pick up things right. Try that before assuming something else is broken. You may also need to restart the laptop entirely...windows tends to get confused easily when rapidly switching networks.

2) Very likely that you'll f up DNS somewhere along the way...so when trying to work out whether a path through the firewall works try to use IPs like 1.1.1.1 with ping rather than google.com.

3) If genuinely stuck do ping device by device. i.e. my laptop is connected to a proxmox device...can i ping that by IP. Yes, ok can I ping opnsense by IP. Yes, can I ping 1.1.1.1. Yes. Can I ping google.com.

Devices

You should point all devices at the opnsense instance for both gateway and DNS [10.32.0.1 for me]. Never point a device at the pihole IP directly...that would work yes but it'll get you into trouble later in unexpected & hard to troubleshoot ways. You want the devices asking opnsense for DNS and opnsense in turn asking the pihole/adguardhome, and that asking 1.1.1.1 in turn. It's under Unbound in opnsense...section called Override...that's where the pihole goes. Also I suggest using Adguard home...it's superior to pihole.

YAY & Thoughts

Once it all sorta works, export the config in opnsense and save a copy of the proxmox /etc/network/interfaces - opnsense has a really nice "one config backup file everything is in it" config thing going...use it.

If anyone finds holes in this post please do tell...I'm running this live so if I'm wrong about something I'd like to find out from you rather than an evil hacker

Full /etc/network/interfaces for completeness

root@firewall:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface enp2s0 inet manual

iface enp3s0 inet manual
iface enp4s0 inet manual
iface enp5s0 inet manual
iface enp6s0 inet manual
iface enp7s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.32.0.2/24
        gateway 10.32.0.1
        bridge-ports enp7s0 enp6s0 enp5s0 enp4s0 enp3s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.31.0.5/24
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

And firewall view:

Introduction

Back on May 19, 2023, the 大 (big, great) @tang_cn wrote an awesome LES tutorial called How to Become an One-Man NAT VPS Provider?

@tang_cn's tutorial covers setting up NAT VPSes using the virt-install command line utility from libvirt.

Trying On A Debian 12.2 Node

I decided to try following @tang_cn's tutorial. The Node which @tang_cn's tutorial had used was running Debian 11. I tried with a Debian 12.2 Node. On Debian 12.2, I ran into two issues:

• @tang_cn's example code did not include a possibly newly required line --osinfo detect=on,name=OSNAME. Adding the osinfo detect=on,name=debian12 line resulted in a fatal error. So, yes, Debian is great, but Debian 12.2's virt-install did not include "debian12" among the list of supported releases shown by virt-install --osinfo list! I bypassed the error with --osinfo detect=on,require=off, but I am unclear on what, if any problems turning off osinfo require might have caused.

• After completing @tang_cn's install steps, my Debian guest VM worked fine over remote VNC via ssh tunnel into the VM's console. The VM could ping and could be pinged from the WAN over IPv6, but not over IPv4. I didn't solve the NAT IPv4 networking issue, whatever it was.

Trying On A Fedora Rawhide Node

I tried again on a Node runniing Fedora Rawhide. I did find that the more current but much less tested code in Rawhide enabled virt-install to support Debian 12 VMs.

• The Rawhide Node

The Rawhide Node is an Intel E-2276G rented from the Hetzner server auction for €33.70 monthly. This price is at least a couple of Euros above the minimum auction price for this server. Nevertheless, this server is worth keeping long term since it has two new NVMe disks. Not all auction servers have new disks.

• References

Fedora has an introductory Virtualization – Getting Started tutorial. The Fedora tutorial links to libvirt's wiki article on common libvirt networking configurations.

Let's start by following the command line versions of the steps in the Fedora tutorial.

• Step 1: System Requirements

Virtualization requires certain minimum RAM amounts and also CPU virtualization extensions, either Intel VT or AMD-V. The minimum RAM amounts are satisfied here since the Node has 64 GB DDR4 ECC RAM. The CPU passes the wiki article's egrep '^flags.*(vmx|svm)' /proc/cpuinfo check.

• Step 2: Install Virtualization Software

The Fedora dnf package manager's virtualization group on our Rawhide server still seems the same as that shown in the Fedora wiki.

[root@polonium ~]# dnf groupinfo virtualization
Last metadata expiration check: 3:24:19 ago on Sun 19 Nov 2023 03:19:10 PM UTC.
Group: Virtualization
 Description: These packages provide a graphical virtualization environment.
 Mandatory Packages:
   virt-install
 Default Packages:
   libvirt-daemon-config-network
   libvirt-daemon-kvm
   qemu-kvm
   virt-manager
   virt-viewer
 Optional Packages:
   guestfs-tools
   python3-libguestfs
   virt-top
[root@polonium ~]# 

Let's try installing with the optional packages.

dnf group install --with-optional virtualization

Whew! That was 481 packages, but the install took only about one minute. The entire 2,130 lines of terminal output was saved, so, if you want to see it, please ask.

Let's use systemctl to start and then to enable the libvirt daemon. Enabling the daemon makes it restart automatically when the Node reboots.

[root@polonium ~]# date -u; systemctl start libvirtd
Sun Nov 19 10:49:12 PM UTC 2023
[root@polonium ~]# date -u; systemctl enable libvirtd
Sun Nov 19 10:49:30 PM UTC 2023
Created symlink /etc/systemd/system/multi-user.target.wants/libvirtd.service → /usr/lib/systemd/system/libvirtd.service.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd.socket → /usr/lib/systemd/system/libvirtd.socket.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd-ro.socket → /usr/lib/systemd/system/libvirtd-ro.socket.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd-admin.socket → /usr/lib/systemd/system/libvirtd-admin.socket.
[root@polonium ~]# 

Verify that KVM is properly loaded.

[root@polonium ~]# lsmod | grep kvm
kvm_intel             425984  0
kvm                  1376256  1 kvm_intel
irqbypass              12288  1 kvm
[root@polonium ~]# 

• Step 3: Default Networking Support

The Fedora tutorial tells us to expect a 192.168.x.x subnet to be set up by default on the Node. The VM guests should be able to connect out to the WAN by using the Node's main IPv4 as a gateway. Connections from the WAN into the guests require firewall DNAT rules on the Node or a bridged environment.

Let's use the iproute2 suite to see what happened to the default network when we installed the dnf virtualization group.

[root@polonium ~]# ip link show
  [ . . . ]
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
[root@polonium ~]# 

It's apparent that installing the virtualization group added a bridge called virbr0. The bridge is down now, but will be put up when we install and start a VM.

• Step 4: Create A Guest With virt-install

Let's install a Debian guest VM. Even though we are following a Fedora tutorial. 🙈🙉🙊

First we get the guest's iso image, SHA512 sum, and signature. The Fedora tutorial put the iso images in /var/lib/libvirt/images/. I began by creating the images directory and then a separate iso directory inside /var/lib/libvirt/images. Libvirt seems flexible about the iso location. Because I use the iso images for additional purposes beyond libvirt, if I were doing this again, I might create the iso directory in /var/lib, independent of and on the same level as the libvirt directory.

Reference: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/

cd /var/lib/libvirt/images/iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.2.0-amd64-netinst.iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS.sign

Next, let's check the SHA512 sum and the signature.

Reference: https://www.debian.org/CD/verify

sha512sum -c SHA512SUMS --ignore-missing
gpg --keyserver keyring.debian.org --recv DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg --verify SHA512SUMS.sign SHA512SUMS

Now we get to use virt-install to create our Debian 12 guest. Here is the install command directly from the Fedora tutorial.

virt-install --name Fedora39 \
--description 'Fedora 39 Workstation' \
--ram 4096 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/Fedora-Workstation-39/Fedora-Workstation-39-20180518.0.x86_64.qcow2,size=20 \
--os-type linux \
--os-variant fedora39 \
--network bridge=virbr0 \
--graphics vnc,listen=127.0.0.1,port=5901 \
--cdrom /var/lib/libvirt/images/Fedora-Workstation-39/Fedora-Workstation-Live-x86-64-39-1.1.iso \
--noautoconsole

Here is the virt-install command which was used for this test. Note that --os-type has been deprecated, and that the version of virt-install, still numbered 4.1.0 in Rawhide, supports Debian 12.

[root@polonium ~]# cat -n /root/Install-Debian-KVM-Guest.sh
     1  virt-install --name Debian-12-Guest-1 \
     2  --description 'Debian 12 from netinst.iso' \
     3  --ram 4096 \
     4  --vcpus 2 \
     5  --disk path=/var/lib/libvirt/images/Debian-12-20231119.1.x86_64.qcow2,size=20 \
     6  --os-variant debian12 \
     7  --network bridge=virbr0 \
     8  --graphics vnc,listen=127.0.0.1,port=5901 \
     9  --cdrom /var/lib/libvirt/images/iso/debian-12.2.0-amd64-netinst.iso \
    10  --noautoconsole
[root@polonium ~]# /root/Install-Debian-KVM-Guest.sh

Starting install...
Allocating 'Debian-12-20231119.1.x86_64.qcow2'                    |    0 B  00:00:00 ... 
Creating domain...                                                |    0 B  00:00:00     

Domain is still running. Installation may be in progress.
You can reconnect to the console to complete the installation process.
[root@polonium ~]# 

virt-install leaves us with a guest which is running. The guest has a disk image qcow2 file and also is connected to the net-install iso. virt-install boots the guest from the iso. Thus, at this point, we have installed the guest on the Node, but we still need to install the guest OS inside the guest's disk image.

• Step 5: Install The Guest OS Inside The Guest

We connect to and install the guest in the usual way appropriate to the distribution that the guest is using. Inside the guest, the guest's IPv4 network is automatically configured by DHCP using the subnet IP assigned to the guest and using the IP of the Node as the gateway.

To connect to the guest from our local address, we set up an ssh tunnel from our local machine to the Node.

ssh root@$NODE_IP -L 5901:localhost:5901 

Then we connect via VNC. In our local VNC client, we enter the localhost IP address followed by the port number which was used in the virt-install --graphics vnc,listen=127.0.0.1,port=5901 \ flag. So, here, we would connect to 127.0.0.1:5901.

When we are finished installing the guest OS on the guest qcow2 image, we normally remove the iso and click reboot in the installer so that the guest reboots from the qcow2 image rather than from the netinst iso. However, here, with libvirt, the guest shuts down and does not reboot. Instead, we now need to start the guest with virsh.

• Step 6: Manage The Guest With virsh

virt-install leaves an XML file for each guest in /etc/libvirt/qemu.

To start a virtual machine, use virsh create with the full filename of the guest's XML file.

[root@polonium ~]# cd /etc/libvirt/qemu
[root@polonium qemu]# ls -l
total 16
drwx------. 2 root root 4096 Nov  6 00:00 autostart
-rw-------. 1 root root 7235 Nov 20 03:58 Debian-12-Guest-1.xml
drwx------. 3 root root 4096 Nov 19 22:49 networks
[root@polonium qemu]# virsh create Debian-12-Guest-1.xml
Domain 'Debian-12-Guest-1' created from Debian-12-Guest-1.xml

[root@polonium qemu]# virsh list
 Id   Name                State
-----------------------------------
 1    Debian-12-Guest-1   running

[root@polonium qemu]# 

To gracefully stop a guest:

# virsh shutdown <virtual machine (name | id | uuid)>

To destroy a guest which already has been stopped. The guest's qcow2 image needs to be removed manually.

# virsh undefine <virtual machine (name | id | uuid)>

Success!

Logging into the fully installed and rebooted guest via VNC showed IPv4 WAN connectivity from inside the guest.

Also, we can use ssh to log in to the guest from the Node.

[root@polonium ~]# ssh tom@192.168.xxx.xxx
The authenticity of host '192.168.xxx.xxx (192.168.xxx.xxx)' can't be established.
ED25519 key fingerprint is SHA256:0NfbDJkPk5Lt+dsFOlt/tWOq2+Edf/UPn136Yis9Kvo.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.xxx.xxx' (ED25519) to the list of known hosts.
tom@192.168.xxx.xxx's password: 
Linux debian 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Nov 19 22:44:07 2023
tom@debian:~$ 

Note: QEMU/KVM Without libvirt

We could have used qemu without libvirt. But that might not be so easy. Let's see the qemu command that libvirt used.

[root@polonium ~]# ps aux | grep libvirt | grep -v grep
  [ . . . ]
qemu       25641  0.9  0.9 5580140 646992 ?      Sl   04:45   0:27 /usr/bin/qemu-system-x86_64 -name guest=Debian-12-Guest-1,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-Debian-12-Guest-1/master-key.aes"} -machine pc-q35-8.1,usb=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu host,migratable=on -m size=4194304k -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296} -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 2d88e4ec-782f-4da6-961e-0266d08f845a -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=32,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device {"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"} -device {"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"} -device {"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"} -device {"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"} -device {"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"} -device {"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"} -device {"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"} -device {"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"} -device {"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"} -device {"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"} -device {"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"} -device {"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"} -device {"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"} -device {"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"} -device {"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/Debian-12-20231119.1.x86_64.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","driver":"qcow2","file":"libvirt-2-storage","backing":null} -device {"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1} -device {"driver":"ide-cd","bus":"ide.0","id":"sata0-0-0"} -netdev {"type":"tap","fd":"34","vhost":true,"vhostfd":"36","id":"hostnet0"} -device {"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:ec:db:35","bus":"pci.1","addr":"0x0"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev socket,id=charchannel0,fd=31,server=on,wait=off -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"} -device {"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"} -audiodev {"id":"audio1","driver":"none"} -vnc 127.0.0.1:1,audiodev=audio1 -device {"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"} -global ICH9-LPC.noreboot=off -watchdog-action reset -device {"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"} -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device {"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
[root@polonium ~]# 

Note: Command Summary

For cut and paste fiends, here's a list of all the commands run in this post.

# egrep '^flags.*(vmx|svm)' /proc/cpuinfo
dnf group install --with-optional virtualization
systemctl start libvirtd
systemctl enable libvirtd
# lsmod | grep kvm
# ip link show
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.2.0-amd64-netinst.iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/SHA512SUMS.sign
sha512sum -c SHA512SUMS --ignore-missing
gpg --keyserver keyring.debian.org --recv DF9B9C49EAA9298432589D76DA87E80D6294BE9B
gpg --verify SHA512SUMS.sign SHA512SUMS

virt-install --name Debian-12-Guest-1 \
--description 'Debian 12 from netinst.iso' \
--ram 4096 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/Debian-12-20231119.1.x86_64.qcow2,size=20 \
--os-variant debian12 \
--network bridge=virbr0 \
--graphics vnc,listen=127.0.0.1,port=5901 \
--cdrom /var/lib/libvirt/images/iso/debian-12.2.0-amd64-netinst.iso \
--noautoconsole

virsh create Debian-12-Guest-1.xml
# virsh shutdown <virtual machine (name | id | uuid)>
#ps aux | grep qemu | grep -v grep

Future Steps

• Set up IPv4 NAT and IPv6/48 on the Node.
• Add IPv6/64 inside the guest.
• Set up Spice.
• Try other Linux and BSD guests.

Introduction

This article presents background information plus a quick and easy to follow recipe for installing, directly from Github source code repositories, the Python programming language and the Django web framework. The operating system used here is Fedora Rawhide. Also included is a note about using the install steps presented here on Debian 12.

As shown by the above screenshot, the quick, easy to follow, source-based Django install recipe given here actually works on Fedora Rawhide. Of course, the "without optimization" approach taken here is not intended for production. Instead, what we have here is a very fun introduction to building Python and Django from source code.

What Is Django?

Django is a web framework written in Python. The Django Project's landing page describes Django as "a high-level Python web framework that encourages rapid development and clean, pragmatic design."

Why Is Django Interesting?

A while back, I watched one of the CS50 lecture videos about Django. The CS50 video shows how simple and fast the Django web development process looks and how much Django must do in the background (for example, handling both database and web server functions). Web development using Django code seems cool -- very few lines of Django code are required to make dynamic web pages work!

Django Install Methods

At first sight, especially for someone coming without a Python background, installing Django seems really complicated! The Django Project's Quick install guide describes:

• installing Python,
• Optionally setting up a database,
• Three Django install methods
  • from an official Django release (recommended),
  • from your operating system's distribution,
  • from the latest development code, and
• Verifying the install.

The Quick install guide states that the first of the three install methods, installing from an official Django release, is "the best approach for most users."

However, for each of the three install methods, including for the recommended install method, the Django Quick install guide sends its readers to different sections of a significantly more detailed How to install Django page.

The more detailed How to install Django page explains that the recommended install from an official Django release involves both pip, the Package Installer for Python, and venv, the Python Virtual Environment Module.

Of course, for a Django install, Python itself, along with pip and venv might need to be installed if all three of these are not already present.

People use Django with a database. The database frequently is SQLite or PostgreSQL. Precompiled SQLite is installed during the preliminary Fedora development tools install discussed below. Although there is an official Github SQLite source mirror, SQLite uses fossil for source code management. Fossil seems so interesting and fun that it would make a great subject for another article. Thus, and even though PostgreSQL also is enticing, for the time being, on Fedora Rawhide, we can proceed with the distribution-installed SQLite.

Besides a database, Django also needs a web server. The official Django How to install tutorial recommends Apache web server with mod_wsgi. The web server install also might be another great subject for yet another LES Talk article. Meanwhile, as shown below, we can use Django's built-in Python web server. It is the built-in Python web server which served the page of which the above screenshot was taken.

Reading the Django Quick install guide was super helpful for important context, and reading the more detailed Django How to install guide was even more helpful for learning additional install details. But, the crux is, even after reading both guides, no specific install recipe was provided. Instead, readers of these well meant and helpful guides are in the end left facing a complex puzzle about how to choose among various install methods and sub-methods. Ironically, each of the install methods very kindly is intended to make installing Django easy!

Simplifying The Django Install

Having tried all three of the install methods listed in the official Django Quick install guide, and having tried several variations mentioned in the more detailed, official Django How to install guide, I wondered whether the Django install somehow could be easier. I wanted a quick Django install recipe!

Many source repositories have easy to find, basic configuration and compilation instructions. For example, Python's simple, standard build instructions are here. Often the simple, standard, "configure, make, make test, make install" series just works. Trying a source-based install seemed faster and easier, closer to upstream, and more fun than figuring out a specific install recipe based on one among the multiple, non-source-based install methods discussed in the official guides.

Source-based installs require a compiler and other development tools. In an ideal world, both the compiler and the development tools also themselves would be compiled from source by the end user. However, as shown here below, the Fedora distribution's package system can be used to quickly and easily install a compiler and other development tools.

Therefore, because of the simplicity of the standard build instructions and the ease of installing a compiler, I attempted, on Fedora Rawhide, a quick, source-based Django install. It seemed a good idea also to compile Python from C and to add pip and venv from their current development Python sources. That way, we get Django's friends built from their current sources to accompany our current, source-built Django. Of course, Fedora Rawhide runs on current Linux development kernels, so Rawhide might be an especially fun and good fit for current, source-built userland programs.

Clone, compile, test, and install times mentioned below are from an Intel E3-1275 v5 with 64 GB DDR4 ECC RAM and 2 x 512 GB NVMe disks rented from Hetzner for €26.70 per month (approximately US$28.34). Newer and faster processors with faster memory and faster disks will have shorter times.

Quick Source-Based Django Install Recipe

Part I: Install Development Tools On Fedora Rawhide

[not_oles@radium ~]$ sudo dnf groupinstall --with-optional "C Development Tools and Libraries"
[not_oles@radium ~]$ sudo dnf groupinstall --with-optional "Development Tools"
[not_oles@radium ~]$ sudo dnf groupinstall --with-optional "Development Libraries"

That's all, only these three commands. Probably they could be combined into a long, one line command, but I wanted to pause long enough between commands to read the individual output of each.

These three commands do not install Django, but they do install, respectively, 250, 585, and 186 development related software packages. That's 1,021 packages! Most of the newly installed packages are unneeded for our present purpose, but it was quick and easy to install everything. The entire one thousand plus package install took only about 8 minutes!

At least some Python must have been present in Fedora prior to the development tools install because dnf, the Fedora install tool, is itself Python based. Therefore, we need a trick to separate dnf installed programs from our own source-built programs. The PATH environment variable can help keep the dnf and source-built programs separated.

Part II: Check PATH And Enable Revert

• PATH

We need to check that both our root and our user PATH environment variables are set for our shell to find our newly installed /usr/local/bin binaries ahead of previously installed package system binaries. Fedora's PATH was okay because /usr/local/bin comes ahead of /usr/bin. Our install won't put anything in /usr/local/sbin, but for additional local installs from source, it's good for usr/local/sbin to come ahead of /usr/sbin.

[root@radium ~]# echo $PATH
/root/.local/bin:/root/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
[root@radium ~]# 

[not_oles@radium ~]$ echo $PATH
/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
[not_oles@radium ~]$

• Revert

Let's set up the ability to revert. We're going to be installing into /usr/local. So, in case we want to revert, we can use tar to back up /usr/local as it is prior to our install.

If we have enough disk space, there's no need for compressing our tar archive. Compression can take significant extra time.

tar is an old program! Old enough not to need the "-" before the flags!

[not_oles@radium ~]$ cd /usr
[not_oles@radium usr]$ sudo tar cvf local-revert.tar local

Part III: Install Python From Source

[not_oles@radium ~]$ cd /usr/local/src
[not_oles@radium src]$ time sudo git clone https://github.com/python/cpython.git
[not_oles@radium src]$ sudo mkdir cpython-obj
[not_oles@radium src]$ cd cpython-obj/
[not_oles@radium cpython-obj]$ time sudo ../cpython/configure --prefix="/usr/local" 
[not_oles@radium cpython-obj]$ time sudo make -j `nproc`
[not_oles@radium cpython-obj]$ time sudo make -j `nproc` test
[not_oles@radium cpython-obj]$ time sudo make -j `nproc` install

The above configure, make, make test, and make install procedure installs pip, as seen in the following snippet from the make install output.

Successfully installed pip-23.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour 
with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

The newly compiled and installed items are:

[not_oles@radium ~]$ ls -l /usr/local/bin
total 28408
lrwxrwxrwx. 1 root root        8 Oct 15 22:09 idle3 -> idle3.13
-rwxr-xr-x. 1 root root      100 Oct 15 22:08 idle3.13
-rwxr-xr-x. 1 root root      230 Oct 15 17:40 pip3
-rwxr-xr-x. 1 root root      230 Oct 15 17:40 pip3.13
lrwxrwxrwx. 1 root root        9 Oct 15 22:09 pydoc3 -> pydoc3.13
-rwxr-xr-x. 1 root root       85 Oct 15 22:08 pydoc3.13
lrwxrwxrwx. 1 root root       10 Oct 15 22:09 python3 -> python3.13
-rwxr-xr-x. 1 root root 29065592 Oct 15 22:08 python3.13
-rwxr-xr-x. 1 root root     3026 Oct 15 22:08 python3.13-config
lrwxrwxrwx. 1 root root       17 Oct 15 22:09 python3-config -> python3.13-config
[not_oles@radium ~]$ 

Note that the pip described in the warning as successfully installed was installed as pip3 and as pip3.13, but not as plain pip. For Python itself, similar to pip, pip3, and pip3.13, the above configure, make, make test, and make install procedure installed python3.13 and python3 as a symbolic link to python3.13, but not python. sqlite3 also is not installed.

On Fedora Rawhide, the complete clone, configure, make, make test, and make install Python procedure took 9 minutes and 51 seconds.

Part IV: Install nox From Source

nox is a requirement for installing pip from source. The pip development Getting Started page says to install nox inside a virtual environment. However, the nox Github readme.md says, "Nox is designed to be installed globally (not in a project virtual environment)." On Fedora Rawhide, both types of installs seem initially to work. Here is the global nox install.

[not_oles@radium ~]$ cd /usr/local/src
[not_oles@radium src]$ time sudo git clone https://github.com/wntrblm/nox.git
[not_oles@radium src]$ time sudo cd nox
[not_oles@radium nox]$ time sudo python3 -m pip install -e . # Note final posix dot.

Part V: Create, Activate, And Enter A Python Virtual Environment

[not_oles@radium ~]$ time python3 -m venv .venv
[not_oles@radium ~]$ source .venv/bin/activate
(.venv) [not_oles@radium ~]$ cd .venv
(.venv) [not_oles@radium .venv]$

Part VI: Install pip From Source In The Virtual Environment

# Reference:  https://pip.pypa.io/en/latest/development/getting-started/
(.venv) [not_oles@radium .venv]$ time git clone https://github.com/pypa/pip
(.venv) [not_oles@radium .venv]$ cd pip
(.venv) [not_oles@radium pip]$ time python3 -m pip install -e . # Note final posix dot.

Part VII: Install Django From Source In The Virtual Environment

# Reference: https://docs.djangoproject.com/en/4.2/topics/install/#installing-development-version
(.venv) [not_oles@radium .venv]$ time git clone https://github.com/django/django.git
(.venv) [not_oles@radium .venv]$ cd django/
(.venv) [not_oles@radium django]$ time python3 -m pip install -e .

Part VIII: How To Exit The Virtual Environment

(.venv) [not_oles@radium django]$ deactivate
[not_oles@radium django]$ cd
[not_oles@radium ~]$ 

Part IX: Test

• Normal Environment

[not_oles@radium ~]$ which python
/usr/bin/python
[not_oles@radium ~]$ /usr/bin/python -V
Python 3.12.0
[not_oles@radium ~]$ which python3
/usr/local/bin/python3
[not_oles@radium ~]$ /usr/local/bin/python3 -V
Python 3.13.0a1+
[not_oles@radium ~]$ which nox
/usr/local/bin/nox
[not_oles@radium ~]$ /usr/local/bin/nox --version
2023.4.22
[not_oles@radium ~]$ which pip
/usr/bin/pip
[not_oles@radium ~]$ /usr/bin/pip -V
pip 23.2.1 from /usr/lib/python3.12/site-packages/pip (python 3.12)
[not_oles@radium ~]$ which pip3
/usr/local/bin/pip3
[not_oles@radium ~]$ /usr/local/bin/pip3 -V
pip 23.2.1 from /usr/local/lib/python3.13/site-packages/pip (python 3.13
[not_oles@radium ~]$ which django-admin # PATH in global environment
/usr/bin/which: no django-admin in (/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
[not_oles@radium ~]$ which sqlite
/usr/bin/which: no sqlite in (/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
[not_oles@radium ~]$ which sqlite3
/usr/bin/sqlite3
[not_oles@radium ~]$ sqlite3 -version
3.43.1 2023-09-11 12:01:27 2d3a40c05c49e1a49264912b1a05bc2143ac0e7c3df588276ce80a4cbc9balt1 (64-bit)
[not_oles@radium ~]$ uname -r
6.6.0-0.rc5.20231013git10a6e5feccb8.44.fc40.x86_64
[not_oles@radium ~]$ cat /etc/fedora-release 
Fedora release 40 (Rawhide)
[not_oles@radium ~]$ 

• Virtual Environment

(.venv) [not_oles@radium .venv]$ which python
~/.venv/bin/python
(.venv) [not_oles@radium .venv]$ python -V
Python 3.13.0a1+
(.venv) [not_oles@radium .venv]$ which python3
~/.venv/bin/python3
(.venv) [not_oles@radium .venv]$ python3 -V
Python 3.13.0a1+
(.venv) [not_oles@radium .venv]$ which nox
/usr/local/bin/nox
(.venv) [not_oles@radium .venv]$ nox --version
2023.4.22
(.venv) [not_oles@radium .venv]$ which pip
~/.venv/bin/pip
(.venv) [not_oles@radium .venv]$ pip --version
pip 24.0.dev0 from /home/not_oles/.venv/pip/src/pip (python 3.13)
(.venv) [not_oles@radium .venv]$ which pip3
~/.venv/bin/pip3
(.venv) [not_oles@radium .venv]$ pip3 --version
pip 24.0.dev0 from /home/not_oles/.venv/pip/src/pip (python 3.13)
(.venv) [not_oles@radium .venv]$ which django-admin
~/.venv/bin/django-admin
(.venv) [not_oles@radium .venv]$ django-admin --version
5.1.dev20231015082711
(.venv) [not_oles@radium .venv]$ which sqlite # Get PATH inside virtual environment :)
/usr/bin/which: no sqlite in (/home/not_oles/.venv/bin:/home/not_oles/.local/bin:/home/not_oles/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
(.venv) [not_oles@radium .venv]$ which sqlite3
/usr/bin/sqlite3
(.venv) [not_oles@radium .venv]$ sqlite3 -version
3.43.1 2023-09-11 12:01:27 2d3a40c05c49e1a49264912b1a05bc2143ac0e7c3df588276ce80a4cbc9balt1 (64-bit)
(.venv) [not_oles@radium .venv]$ uname -r
6.6.0-0.rc5.20231013git10a6e5feccb8.44.fc40.x86_64
(.venv) [not_oles@radium .venv]$ cat /etc/fedora-release 
Fedora release 40 (Rawhide)
(.venv) [not_oles@radium .venv]$ 

• Web Server

We can launch a web server, and then our browser can see the Django default html index page shown in the screenshot above!

[not_oles@radium ~]$ cd .venv
[not_oles@radium .venv]$ source bin/activate
(.venv) [not_oles@radium .venv]$ time django-admin startproject mysite
(.venv) [not_oles@radium .venv]$ cd mysite/
(.venv) [not_oles@radium mysite]$ python3 manage.py runserver 127.0.0.1:42365

Time Summary

Debian 12

Newly installed out of its netinstall.iso box, Debian 12 Linux seems to use /usr/local/lib for apt Python installations. In other words, /usr/local/lib comes with subdirectories /python3.11 and /python3.11/dist-packages preinstalled.

Moreover, Debian valiantly guards /usr/local. The above CPython "configure, compile, test, install" command sequence does not install any pip. Also, on Debian 12, the above nox install command failed. There was a helpful error message, which explained, "This environment" [that is, I guess, global /usr/local] "is externally managed. . . ."

Discussion

• What are the advantages and disadvantages of installing nox globally versus installing nox inside a virtual environment?
• Is using python for the packaged install and python3 for the compiled install a good idea?
• Should we do something like sudo ln -s /usr/local/bin/python3.13 /usr/local/bin/python?
• CPython has a helpful Setup and Building Guide. In addition to configuring and compiling, git, dependencies, several operating systems, and troubleshooting also are discussed. The Guide says, "There is normally no need to install your built copy of Python! The interpreter will realize where it is being run from and thus use the files found in the working copy."
• I need to study up on the cpython configure options, including --with-pydebug and --enable-optimizations.
• The Cpython test_dtrace formerly succeeded on Rawhide, but it has failed since about the time earlier this month when Python 3.12 was released. There is an old bug report from 2019 which has similar log messages. Configuring Cpython with --with-dtrace did not help.

Also note that here I'm downloading a fp16 model and converting it to q8 GGUF. In practice you can skip over those steps and just download ready made quantized GGUF models from TheBloke's huggingface repo.. i.e. You'd modify the download model step to point to a quantized GGUF model and skip the generate and quantize step after that.

This assumes Ubuntu 22.04 - you may need to do stuff like install python3 if you're on a different distro

Check that we have a GPU

apt update && apt upgrade
apt install hwinfo -y
hwinfo --gfxcard --short

Set up nvidia driver and SDK

apt install nvidia-driver-535-server nvidia-dkms-535-server nvidia-cuda-toolkit -y
reboot
nvidia-smi
nvcc --version

Grab llama.cpp and build it

git clone https://github.com/ggerganov/llama.cpp
apt install cmake -y
cd llama.cpp
mkdir build
cd build
cmake .. -DLLAMA_CUBLAS=ON
cmake --build . --config Release
cd ..

Download a model

mkdir -p /root/llama.cpp/models/llama2-fp16
python3 -m pip install huggingface_hub
python3
from huggingface_hub import snapshot_download
snapshot_download(repo_id="TheBloke/Llama-2-13B-Chat-fp16", revision="main",local_dir="/root/llama.cpp/models/llama2-fp16/")
quit()

Generate GGUF file

python3 -m pip install gguf sentencepiece
python3 convert.py ./models/llama2-fp16/

Quantize it

cd ./build/bin
./quantize ../../models/llama2-fp16/ggml-model-f16.gguf ../../models/llama2-q8.gguf q8_0

Run it

./main -m ../../models/llama2-q8.gguf -ngl 99 --color -p "Tell me a story about a unicorn!"

Tell me a story about a unicorn!

Once upon a time, in a far-off land of rolling hills and sparkling streams, there lived a beautiful unicorn named Luna. She had a shimmering coat of silver and white, and her horn was as bright as the stars in the night sky.

Luna lived a peaceful life, roaming the forests and meadows, and making friends with all the creatures she met. She loved to play with the butterflies and dance with the flowers, and she could make the most beautiful music with her horn.

One day, a wicked witch cast a spell on the land, causing all the plants and animals to become sick and tired. The unicorns were especially affected, and their beautiful coats became dull and lifeless.

Luna knew that she had to do something to save her friends and the land they lived in. She set out on a journey to find the witch and break her spell.

As she traveled through the forest, Luna met many creatures who were suffering from the witch's spell. She used her horn to heal them and bring them back to life. She also met a brave knight who had been searching for the witch for many years. Together, they journeyed on, determined to defeat the wicked witch and bring peace back to the land.

Finally, after many days of traveling, they came to the witch's castle. It was a dark and gloomy place, surrounded by a moat of swirling black water. But Luna was not afraid. She knew that her horn could break any spell, no matter how powerful.

She and the knight entered the castle, ready to face whatever dangers lay inside. As they made their way deeper into the castle, they came across the witch herself. She was a terrifying sight, with warts and a crooked nose, and a cackle that sent chills down your spine.

But Luna was not afraid. She raised her horn and pointed it at the witch, ready to break the spell. The witch laughed and tried to stop her, but Luna's horn was too powerful. With one blast of magic, the spell was broken, and the land was once again filled with light and life.

The creatures who had been turned to stone were returned to their true forms, and they cheered and celebrated as Luna and the knight emerged from the castle. The witch was banished from the land forever, and peace was restored.

And Luna, the little unicorn with the powerful horn, lived happily ever after, knowing that she had saved her homeland from the evil witch's spell. The end.

For over four years now I have been enjoying a Google Cloud Platform (GCP) Free Tier Debian VPS. During this time, Debian's package system cache has grown to use more than half of the VPS's 10 GB of file storage space to cache .deb package files. This post explains the history and the use of this wonderful, free VPS (Thanks, Google!); shows the size of the package cache; and explains how to reclaim the large amount of disk space used for the cache.

VPS Creation in 2019

According to the Google Cloud Console, my e2-micro instance was created on January 27, 2019. That was more than four and a half years ago!

Larger File System Size Possible Within Current GCP Free Tier Limits

Looking at the current GCP Free Tier Resource Limits suggests that the Compute Engine limit is "30 GB-months standard persistent disk." My VPS is only 10 GB, so maybe, when I need it, I could increase the file system size to add an additional 20 GB.

VPS Use

This GCP VPS has been in modest, but continuous, steady use throughout more than four years. It serves my tiny personal website, holds a few small backup files, and runs TinyProxy. Tinyproxy works great for accessing certain websites in the US when I am in Mexico.

I log in to the VPS frequently. I wouldn't say every week for sure, but, if I am working on something, I log in many times in one week. Of course, to update the VPS's Debian operating system, I regularly run apt-get update and then apt-get upgrade.

Running EOL Debian 10 Buster

My GCP VPS is still running Debian 10 Buster (End of Life (EOL) a year ago, September 10, 2022), which might be okay, because the check-support-status utility doesn't list any installed packages without Long Term Support (LTS).

root@gc:~# man check-support-status
root@gc:~# check-support-status
root@gc:~# echo $?
0
root@gc:~# 

Great VPS Uptime!

Usually I run the w command when I log in. Here's what I saw today. 590 days of uptime isn't bad!

root@gc:~# w
 20:43:01 up 590 days, 16:55,  1 user,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
tom      pts/0    187.189.238.2    19:30    0.00s  0.08s  0.01s sshd: tom [priv]    
root@gc:~# 

Low Free Disk Space

One thing I have noticed is that free disk space on the VPS has been decreasing. Here is what free disk space looked like earlier today.

root@gc:~# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       9.8G  8.7G  673M  93% /
root@gc:~# 

The VPS image was set up without a swap partition. Since I might want to add more website or backup files, install additional utilities, or even add a swap file, I thought the remaining 673 MB might be a little small. I decided to take a look around and see if I could find a place or a few places where I could remove a big file that no longer was needed.

root@gc:/# du -sh /home/tom /var/www
77M     /home/tom
67M     /var/www
root@gc:/# 

Hmm! It didn't look like my home directory or even my website files were too piggy! Looking some more, I was surprised to find that /var was using a lot of space!

root@gc:/# du -sh /var
5.9G    /var
root@gc:/# 

Wow! "Lots of logs?" I wondered.

root@gc:/# du -sh /var/*
1.2M    /var/backups
5.7G    /var/cache
156M    /var/lib
4.0K    /var/local
0       /var/lock
13M     /var/log
3.3M    /var/mail
4.0K    /var/opt
0       /var/run
8.7M    /var/spool
12K     /var/tmp
67M     /var/www
root@gc:/# 

What's using all that space in /var/cache?

root@gc:/# du -sh /var/cache/*
108K    /var/cache/apparmor
5.7G    /var/cache/apt
3.8M    /var/cache/debconf
20K     /var/cache/ldconfig
2.5M    /var/cache/man
4.0K    /var/cache/private
root@gc:/# 

Oh! It's the package system! Four years of .deb files live there! 5.7 GB! Looking at the full output of the below command without the pipe to wc, it was instantly clear that all except the first five of the 346 output lines were .deb files. That's 341 .deb files! Together, the 341 cached .deb files used more than half of my VPS's disk space!

root@gc:/# ls -l /var/cache/apt/* | wc
    346    3090   28173
root@gc:/# 

About .deb Files

Binary .deb files are packages containing binaries which are "directly useable by dpkg." Source .deb files contain source code.

Luckily for small VPSes, after the system has been updated, the .deb files may be removed.

How To Reclaim Disk Space

From man apt-get:

       clean
           clean clears out the local repository of retrieved package files. It removes
           everything but the lock file from /var/cache/apt/archives/ and
           /var/cache/apt/archives/partial/.

After running apt-get clean the VPS got its empty disk space back! 6.4 GB available instead of 673 MB!

root@gc:~# apt-get clean
root@gc:~# ls -l /var/cache/apt/*
total 4
-rw-r----- 1 root root    0 Jan 24  2019 lock
drwx------ 2 _apt root 4096 Sep 13 21:22 partial
root@gc:~# du -sh /var/cache/apt
64K     /var/cache/apt
root@gc:~# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       9.8G  3.0G  6.4G  32% /
root@gc:~# 

Disabling The Cache

One possible way to prevent running out of disk space might be to disable the caching of .deb files. From man 5 apt-conf:

DIRECTORIES

       [ . . . ]

       Dir::Cache contains locations pertaining to local cache information, such as the two
       package caches srcpkgcache and pkgcache as well as the location to place downloaded
       archives, Dir::Cache::archives. Generation of caches can be turned off by setting
       pkgcache or srcpkgcache to "".

Please see also this old Bug #753531 from 2014, which was very fun to read because it almost said rm -rf /*!

It seems possibly safer and generally less work all around to leave the .deb caching defaults as they are. It's easy to empty the cache once in awhile with apt-get clean.

Conclusion

For small Debian VPSes which have been in use a long time, you might want to run apt-get clean to reclaim the large amount of disk space used by the apt cache.

0. Advance Notice

• Special thanks to @Not_Oles !
• This tutorial has been verified to work with Debian 11 on Hetzner's dedicated server.
• All commands are executed as root.

• In this tutorial the host's network configuration is:
  Network Interface: eno1
IPv4 Address: 192.0.2.2/26
IPv4 Gateway: 192.0.2.1
IPv6 Address: 2001:db8:ace:babe::1/64
IPv6 Gateway: fe80::1

• We want to create a private network 192.168.0.0/24 for VMs; we want to use 192.168.0.1 as the IPv4 gateway and 2001:db8:ace:babe::1 as the IPv6 gateway for VMs.

• We want to create a VM whose IPv4 address is 192.168.0.2 and IPv6 address is 2001:db8:ace:babe:cafe::1/80.

1. Enable IP Forwarding

Open /etc/sysctl.conf. Find net.ipv4.ip_forward; uncomment this line and set the value to 1. Do the same for net.ipv6.conf.all.forwarding.

Save the file; then run sysctl -p to apply changes.

2. Install Required Packages

apt install qemu-system qemu-utils libvirt-clients libvirt-daemon-system virtinst bridge-utils

3. Modify the Network Configuration

3.1 Make a copy of the original /etc/network/interfaces

cp -p /etc/network/interfaces /etc/network/interfaces.backup

3.2 Open /etc/network/interfaces and edit

Comment out eno1's IPv6 configuration, then add the following line:

iface eno1 inet6 manual

Continue to add the following lines to create an interface br0 for KVM networking:

auto br0
iface br0 inet static
    address 192.168.0.1/24
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
iface br0 inet6 static
    address 2001:db8:ace:babe::1/64
    up ip -6 route add default via fe80::1 dev eno1

bridge ports none means br0 is not attached to any physical interface. bridge_stp off disables Spanning Tree Protocol; usually we don't need it in simple cases. bridge_fd 0 sets the forwarding delay time to 0; 0 is good in simple cases. The iptables line allows LAN nodes with private IP addresses to communicate with external public networks. The ip -6 route line specifies the IPv6 gateway.

Save the file; then run systemctl restart networking.service to apply changes.

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eno1
iface eno1 inet static
    address 192.0.2.2
    netmask 255.255.255.192
    gateway 192.0.2.1
    # route 192.0.2.0/26 via 192.0.2.1
    up route add -net 192.0.2.0 netmask 255.255.255.192 gw 192.0.2.1 dev eno1
iface eno1 inet6 static
    address 2001:db8:ace:babe::1
    netmask 64
    gateway fe80::1

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eno1
iface eno1 inet static
    address 192.0.2.2
    netmask 255.255.255.192
    gateway 192.0.2.1
    # route 192.0.2.0/26 via 192.0.2.1
    up route add -net 192.0.2.0 netmask 255.255.255.192 gw 192.0.2.1 dev eno1
iface eno1 inet6 manual

auto br0
iface br0 inet static
    address 192.168.0.1/24
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o eno1 -j MASQUERADE
iface br0 inet6 static
    address 2001:db8:ace:babe::1/64
    up ip -6 route add default via fe80::1 dev eno1

4. Create the VM

virt-install --name YOUR_VM_NAME --ram MEMORY_SIZE_IN_MB --vcpus=NUMBER_OF_CORES --disk /PATH/TO/VIRTUAL/DISK/IMAGE.qcow2,device=disk,bus=virtio,size=DISK_SIZE_IN_GB,format=qcow2 --graphics vnc,listen=0.0.0.0,port=VNC_PORT,password=VNC_PASSWORD --network bridge=br0 --noautoconsole --cdrom /PATH/TO/ISOFILE.iso --boot cdrom,hd

5. Configure Guest Networking

Because we haven't configured the DHCP service on the host, we have to manually set the VM network settings. If you are using a netinst image to install the OS, then you need to set the IP address to 192.168.0.2/24 and the gateway to 192.168.0.1 during the installation.

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-hotplug ens3
iface ens3 inet static
    address 192.168.0.2/24
    gateway 192.168.0.1
iface ens3 inet6 static
    address 2001:db8:ace:babe:cafe::1/80
    dns-nameservers 2606:4700:4700::1001
    gateway 2001:db8:ace:babe::1

In part one of this guide we will set up an outbound only mail relay server that will connect and send mail via MXRoute.
In part two we will show how to set up sendmail on any VPS that you want to relay mail through the outbound only mail relay server that you set up in part one. You can also use postfix or any other mail service on the secondary servers to relay through the outbound only mail relay server.

NOTE: In an effort to make this rather long guide a more reasonable size I have put many of the code blocks that you will need to copy into display pull downs. You will need to click the to see this information.

Part One

Install required packages

CentOS 7:
yum -y install sendmail sendmail-cf cyrus-sasl cyrus-sasl-plain
yum -y remove postfix

CentOS 8, Alma8linux, Rockylinux:
dnf -y install sendmail sendmail-cf cyrus-sasl cyrus-sasl-plain
dnf -y remove postfix

Make sure your hostname is set

Edit /etc/hosts and ensure there is a line [your IPv4] tab [your host name]
Edit /etc/sysconfig/network and ensure there is a line HOSTNAME=[your host name]

Setup the certificates

Get a cert from Let's Encrypt for your mail relay server (I am not showing how to do this here)
Make a subdirectory called "certs" under /etc/mail/:
mkdir /etc/mail/certs

While in the directory /etc/mail/certs execute the commands:
openssl dhparam -out dhparam.pem 4096
and
ln -s /etc/ssl/certs/ca-bundle.crt ca-bundle.crt
and
chmod 600 dhparam.pem sendmail.pem

You should now have three files in your /etc/mail/certs directory

ca-bundle.crt [a soft link]
dhparam.pem
sendmail.pem

The certificate will need to be undated before it expires.
I should add a script to auto update the mail server cert from LetsEncrypt. (coming soon™)

Configure sendmail.mc

Move the original sendmail.mc to sendmail.mc.org:
mv /etc/mail/sendmail.mc /etc/mail/sendmail.mc.org

After making the above changes execute the command:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Setup MXRoute authinfo

After making the above changes ensure that you are in the directory /etc/mail and execute the command:
makemap hash authinfo < authinfo

Enable sendmail, saslauthd, and reboot

Execute the following commands to enable sendmail and saslauthd to start on boot and then reboot:
/usr/bin/systemctl enable sendmail.service
/usr/bin/systemctl enable saslauthd.service
reboot

Set SPF DNS record

Add the IPv4 address of this mail relay server and MXRoute to your DNS spf record:
TXT "v=spf1 +ip4:[IPv4 address of this mail relay server] include:mxlogin.com -all"

Is it working ?

Test if everything works as expected by running this command:
echo "Subject: hello" | sendmail you@yourdomain
(replace you@yourdomain with your email address)

Check the log file:
cat /var/log/maillog

If you see stat=Sent (OK, you are sending your mail through MXRoute:

Jan 9 16:40:55 f sendmail[2246]: 309KF3K5001390: to=<you@yourdomain>, ctladdr=<root@[your relay mail server]> (0/0), delay=01:25:51, xdelay=00:00:01, mailer=relay, pri=750282, relay=friday.mxlogin.com. [159.69.65.104], dsn=2.0.0, stat=Sent (OK id=1pEzt1-0004Nf-UP)

If you see stat=Deferred, it is not working:

Jan 9 15:30:02 f sendmail[1848]: 309KF3K5001390: to=<you@yourdomain>, ctladdr=<root@[your relay mail server]> (0/0), delay=00:14:58, xdelay=00:00:00, mailer=relay, pri=390282, relay=friday.mxlogin.com., dsn=4.0.0, stat=Deferred

NOTES

If you make any revisions to /etc/mail/sendmail.mc you will need to execute:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

If you make any revisions to /etc/mail/authinfo you will need to execute:
makemap hash authinfo < authinfo

Then restart sendmail:
/usr/bin/systemctl restart sendmail.service

So that the changes you made will take effect.

Are we done yet ?

If you are NOT planning to have any other VMs relay through this mail relay server you are done. All mail from this VM will relay via MXRoute. While sendmail is listening on port 2525 no mail will be relayed except from localhost.

If you DO want to relay other VMs mail through this mail server to also send their mail via MXRoute then continue below.

Allow other VMs to relay through this server

Open /etc/mail/access in your favorite text editor and add each of the VMs that you want to allow mail to relay through this server by adding their IP address, each on a seperate line, at the bottom of the file as Connect:123.123.123.123 RELAY.

After you have added all the VMs you wish to allow to relay to MXRoute via this mail server, and saved the file, execute this command:
makemap hash /etc/mail/access.db < /etc/mail/access

Restart sendmail:
/usr/bin/systemctl restart sendmail.service

You will need to execute the two commands above anytime you make any additions or subtractions to the /etc/mail/access file for the changes to take effect.

Don't forget to open your firewall

To relay mail via your outbound only mail server from other VMs you must open your firewall for port 2525 access.

For firewalld:
firewall-cmd --permanent --zone=public --add-port=2525/tcp
service firewalld stop
service firewalld start

Part Two

In part two we will show how to set up sendmail on any VPS that you want to relay mail through the mail relay server that you set up in part one. It's much easier and faster.

Install required packages

CentOS 7:
yum -y install sendmail sendmail-cf
yum -y remove postfix

CentOS 8, Alma8linux, Rockylinux:
dnf -y install sendmail sendmail-cf
dnf -y remove postfix

Make sure your hostname is set

Edit /etc/hosts and ensure there is a line [your IPv4] tab [your host name]
Edit /etc/sysconfig/network and ensure there is a line HOSTNAME=[your host name]

Edit the sendmail.mc file

Open /etc/mail/sendmail.mc with your favorite text editor and add the lines in the pull down near the bottom of the file between the quoted lines shown above and below the pull down box.

dnl MASQUERADE_DOMAIN(mydomain.lan)dnl

MAILER(smtp)dnl
MAILER(procmail)dnl
dnl MAILER(cyrusv2)dnl

After copying the code to /etc/mail/sendmail.mc you will need to modify the following:
1. Change 123.123.123.123 to the IPv4 of the server that is being installed.
2. Change myrelay.mydomain to the hostname.domainname of your mail relay server from part one.

After making the above changes execute the command:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Enable sendmail and reboot

Execute the following commands to enable sendmail to start on boot and start now :
/usr/bin/systemctl enable sendmail.service
reboot

Set SPF DNS record

If you have not done so previously, add the IPv4 address of the mail relay server and MXRoute to your DNS spf record of the domains for which this VM will be sending mail:
TXT "v=spf1 +ip4:[IPv4 address of mail relay server] include:mxlogin.com -all"

Is it working ?

Test if everything works as expected by running this command:
echo "Subject: hello" | sendmail you@yourdomain
(replace you@yourdomain with your email address)

Check the log file:
cat /var/log/maillog

If you see stat=Sent, and Message accepted for delivery you are sending your mail through your mail relay server:

Jan 10 04:27:12 localhost sendmail[2993586]: STARTTLS=client, relay=my.relayserver., version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Jan 10 04:27:13 localhost sendmail[2993586]: 30A9RCpK2993585: to=you@yourdomain, ctladdr=<root@localhost> (0/0), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=149367, relay=my.relayserver. [IPv4 for my.relayserver], dsn=2.0.0, stat=Sent (30A9RCkP001947 Message accepted for delivery)

On the mail relay server from part one these are the logs of the same email message successfully being relayed through to MXRoute:

Jan 10 04:27:13 my sendmail[1947]: 30A9RCkP001947: from=<root@the1stvps>, size=11346, class=-60, nrcpts=1, msgid=<202301100927.30A9R2jw2993246@the1stvps>, proto=ESMTPS, daemon=MTARelay, relay=the1stvps [IPv4 of 1st vps]
Jan 10 04:27:16 my sendmail[1949]: STARTTLS=client, relay=friday.mxlogin.com., version=TLSv1.2, verify=OK, cipher=ECDHE-RSA-AES128-GCM-SHA256, bits=128/128
Jan 10 04:27:17 my sendmail[1949]: 30A9RCkP001947: to=<you@yourdomain>, delay=00:00:04, xdelay=00:00:04, mailer=relay, pri=239346, relay=friday.mxlogin.com. [159.69.65.104], dsn=2.0.0, stat=Sent (OK id=1pFAuc-0004BM-Uh)

If you see stat=Deferred Connection timed out , it is not working.
In this case the firewall was not open for port 2525 on mail relay server:

Jan 10 04:24:01 localhost sendmail[2987677]: 30A8S7uH2987677: to=<you@yourdomain>, delay=08:45:57, xdelay=00:00:00, mailer=relay, pri=931045, relay=my.mailrelayserver., dsn=4.0.0, stat=Deferred:Connection timed out with my.mailrelayserver.

If you see stat=User unknown , it is not working.
In this case the IPv4 of this server was not in the /etc/mail/access file on the mail relay server:

Jan 10 22:30:47 localhost sendmail[3093806]: 30B3UkCR3093806: to=you@yourdomain, delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=32612, relay=my.relayserver. [IPv4 of my.relaysever], dsn=5.7.1, stat=User unknown

This will be shown on the mail relay server as the following Relaying denied error.

Jan 10 22:30:47 my sendmail[1363]: 30B3UjUR001363: ruleset=check_rcpt, arg1=<you@yourdomain>, relay=the1stvps [IPv4 of 1st VPS], reject=550 5.7.1 <you@yourdomain>... Relaying denied

Are we done yet ?

Yes, for this guide that is it for now.....
But no you are not done yet. At a minumum servers that send outbound mail should use DKIM in addition to having SPF records and all the good stuff that MXRoute provides.

A warning from MXRoute so as to not cause problems

@jarland said:
It’s worth noting that a slight variation on this kind of setup consumes a significant amount of my time trying to protect the infrastructure from being overwhelmed by pure junk that drags down IP reputation by increasing the association between our IPs and spam folders. These are the big line items:

1. Almost no one who does this cares that their hostname isn’t an actual valid domain, forcing me to locate all of their server hostnames and add them to a banned sender list so they get errors instead of flooding the queue with junk I can’t deliver or bounce.
2. That cron job you have set to run every minute sends a notification, and the person who sets this up commonly has all of these emails sent to their Gmail address.
3. A simple command line interface or a cron job notification lacks the required headers that a legitimate email client would add without the user noticing.

Number 1 causes our mail queues to be packed full of “root@vps{1..999}” emails that I can’t and won’t deliver because they’re not from valid envelope senders. But I can’t say that these are inherently bad and prevent them from going out before I identify each one of them because temporary DNS issues could cause a sending address to appear invalid that isn’t later. So I queue them, but then I audit the queues every day for this junk and use it to build my banned sender list (ex. *@localhost, *@vps1, *@vps.local, etc.).

Number 2 loops in with #1, and the two items work hand in hand to try to overly associate our IP addresses with Gmail’s spam folder, and number 3 comes in here as part of that. A bad sending address, a cron job notification that the user never meant for human consumption (and will never read, save for the 1 in a million user who might), and a lack of valid headers combined with Gmail’s threading feature mean you see one little junk email in your Spam folder, but you actually sent 1,440 junk emails per cron job to Gmail that were all destined to the spam folder. Combine that with the 600 other users doing the same thing, and you then have 864,000 emails from our platform daily, which are guaranteed to land in Gmail’s spam folder. Almost none of those users even meant to send those emails. The users rarely have any idea that they sent them, and they have no plan of gaining anything from it. It’s hard to let all of that through and then tell people I’m focused on doing everything I can to ensure inbox delivery.

All this to say, it’s good to be able to configure your systems to do this, but please keep one thing in mind: The MXroute platform is designed to send emails that are intended for either human consumption or automatic processing. It is not intended to be used as a garbage delivery system for Gmail. It also can’t be used as a garbage delivery system for Gmail, expecting that intentionally sent emails have a high probability of landing in the inbox. It can be one or the other. It can’t be both. Doing everything I can to get intentional emails into the inbox has always been my top focus. So please, if you do this, take a moment to consider what your server is doing and how it may use this configuration in ways you weren’t thinking of at the time.

Do make sure that your sending addresses are always valid. If your hostname isn’t valid, has no MX or SPF records itself, then your cron jobs and other system emails are going to be sending as invalid envelope senders. Think “root@localhost” that’s not valid, even “root@myvps.mydomain.tld” is invalid if myvps.mydomain.tld lacks valid MX/SPF. Most of that is fine if you’re sending to yourself to be delivered to an inbox on the same MXroute server that your service is hosted on, but if you’re sending these out to remote recipients that’s when you really need to dot your I’s and cross your T’s, making sure you have everything done properly.

-By A. Vyas, January 2023

Read Part I here: Five Ways to Keep Your Hosting Costs Under Control - Part I

Introduction

This is the second and concluding part of a two part series on how you can reduce your web hosting costs in times of high inflation and increasing web hosting fees. In the previous part, we saw how a person "Lata" was able to save nearly 30 US dollars a month by adopting three different approaches to manage her web hosting plans.

To summarise, the three approaches she adopted so far were:
I. Consolidate the hosting plans
II. Moving the hosting to a cheaper location(s)
III. Lower the specifications a notch or two

In this part, let us explore some more ways that she followed, and determine how much savings she was able to achieve in practical terms.

IV. Check those idlers and duplicates

Idlers are the bane of the Low End aficionados. Lata's case was no different. She had multiple idlers in her webhosting "Empire", and quite a collection of hosting services were in Singapore. From Lata's home location, Singapore offered the greatest set of options and a reasonable ping ~ 45 ms. It was not the fastest or the best latency she could get, but it was much better than locations in Western Europe (~ 150 ms) or the East Coast, USA (~ 220 ms).

Over the past couple of years Lata kept subscribing to newer plans in Singapore, it almost became a habit of sorts. Recently, she decided to cut the number of plans to half. Gone was a small server, a larger KVM, and a reseller hosting plan. All three were mostly idling.

Lata also had multiple KVM plans in Western Europe. Some could be transferred out, but the renewal for these plans was only a few weeks away. She set them to be terminated at the conclusion of their respective periods.

Scheduling to terminate the service at the end of the billing period also gave her an opportunity to backup and migrate her data in a planned manner, instead of rushing things at the last minute.

Tip: You can consider transferring out the plans out to other users in the LE community. However, the level of effort and transfer fees in many cases may outweigh the savings.

Estimated savings till March 2023 : about 30 US Dollars

Tip: Do you have a few extra idlers that are higher than your comfort level? Do you see an opportunity to cut down on a few plans? A great place to start would be to check if you have multiple plans in the same location.

V. Check the pricing terms, mode of payment, and currency

Lata, who is based in Bengaluru in India, was paying for most services in US Dollars through PayPal. She began to explore providers who accepted payment in Indian Rupees (INR). The reason for preferring PayPal will be highlighted in a different post, since it involves a deep dive into regulation, tax implications, as well as other technicalities.

She switched some of her services to providers who accepted payment in INR, and paid them using services such as Razorpay. Paying for longer terms (6 months or higher) helped her and the web host save some percentage points in processing fees as well.

The downside of this approach is that sometimes, this method may not be practical.

For example, Lata had VPS plan with one host from India, for a VPS located in Mumbai. Their monthly charges were 4 US dollars. For annual payment, they offered a 8 dollar discount, that is 40 US Dollars per year. However, Lata was required to pay in US Dollars.

Typically, INR: US Dollar exchange rate is between 70 and 72 (i.e. 70 INR = 1 US Dollar). In recent months, it is hovering closer to INR 80-83 range. Adding Paypal fees, etc- the rate climbs to 85 or 86 INR per US Dollar. In other words, Paypal adds an overhead that negates any potential cost savings. In Lata's case, it resulted in savings of only 2 US Dollars. She dropped the idea of switching the currency and payment terms altogether, and decided to opt for monthly subscription.

Estimated savings till March 2023: Nil

Tip: Are you paying month to month, or a slightly longer term? Does your host offer discounts if you pay for six months or annually? Many hosts, in fact, do!

The savings add up

Needless to say, the steps taken by Lata helped her save some money. In the first part of this series, we saw how Lata was able to save 87 US dollars. In this part, we saw that her proactive steps yielded savings of another 30 US Dollars.

All in all around 120 US dollars saved upto March 2023, or about 40 US dollars a month, is quite a bit of money saved! Maybe she should get a hot, tempting hosting plan during a future sale this year to reward herself for her efforts. Just kidding.

Wrapping it all up

Under normal circumstances, a cost increase due to inflation is expected and mostly a given. Inflation may vary from low single digits to high teens, depending on the country in which one lives. But present times are challenging and dynamic, where double digit inflation is prevalent in many countries at the same time, even those who have not seen such costs in 20 or even 30 years!

Add to this the falling value of the British Pound, or the Euro, in comparison to the US Dollar. I already mentioned INR's fall compared to the US Dollar earlier. The story may not be too different where you live.

But these times also present an opportunity to reflect on the hosting situation and think of ways to keep a tab on the hosting costs. I hope Lata's approach gave you an idea or two. However, the story does not end there.

Some questions that may arise such as:

• Has this post covered most of the practical approaches? (i.e. ones that can be considered as low hanging fruits)?

• Did Lata miss something obvious?

• Can she optimize the costs further?

I leave you with these questions as food for thought, and may the Low End Spirit in you figure out the best way(s) for you.

Author's Note: Thanks to the two reviewers, who shall remain unnamed, for their feedback and suggestions for this post.

-By A. Vyas, January 2023

(This is a two part series that is aimed to help customers looking to optimize their web hosting costs, although many of the suggestions may apply to the web hosts themselves)

Introduction

It is early January 2023 as I write this post. The 2022 end of the year Holiday season has just concluded, the Black Friday/ Cyber Monday and Christmas/ New Year Sales have concluded, and hopefully many of you still have some money left over to pay for your renewing your web hosting plans! This period becomes allows some time to reflect on how much one is spending on subscriptions in general and web hosting services in particular. The possibility of an economic downturn is one reason. The other factor, which I have mentioned below, is already a reality.

Over the past few months, many web hosting providers have increased their hosting fees on account of rising costs. European countries in particular have witnessed very high energy costs, which has added to the problem of high inflation. This has resulted in higher costs for running their business. In most cases, they have no choice other than passing on these increases to their customers.

In such a situation, how can a user, who is say a typical customer of Low End hosting, keep the hosting costs low? Let us take a look at five different ways to help you keep a tight leash over your hosting costs in the coming months.

Web Hosting Plans- The Problem of Plenty

Before we get down to business, let us try and answer this all important question:

Do you have too many hosting plans, or do you have a large collection of hosting subscriptions?

If the answer is yes, you are in a perfect position to take a deep, hard look at your respective hosting collections. There may be some impulse purchases, and some plans that do not make any sense now. Those are ripe to be dropped or culled. For example,

Is there a 2 GB shared hosting plan that you purchased during last Black Friday? Or an older OVZ server that is still active?

If the answer is no, you are in a good position to start optimizing further- maybe switch a plan or two to a lower tier. We will discuss that option in Part II of this series.

Where and How to begin reducing your web hosting costs?

Lata's Approach to Saving on Web Hosting

Let us take a look at the Web Hosting "Empire" of a person called Lata. She has gathered quite a number of webhosting plans over the years. When she was thinking of reducing these expenses she adopted the following approach.

First, she prepared a chart that lists all the hosting plans she actively subscribed to. In Lata's case, as might be the situation with many of you, this presented quite a collection of Shared hosting, Reseller, NAT, OVZ, KVM and Storage servers. In addition, she also has subscriptions for many SaaS (Software as a Service) based hosting plans from providers like Jimdo and Shopify.

In other words, short of a Dedicated server, Lata had subscribed to practically every sort of hosting plan one may typically find here on Lowend Spirit(https://lowendspirit.com/). While some of the hosting plans have been procured for friends and family, others are for her customers. Finally, a few plans are for her own personal or hobby projects.

_{Lata's Webhosting collection}

Let us see how Lata began to arrive at optimizing her costs.You can also develop a method that suits your situation the best. Lata tried five different approaches to arrive at some cost savings. In this first part of the series, we will look at three approaches. The balance will be covered in the second installment.

For sake of simplicity, all costs are mentioned in US Dollar terms, and any savings are projected to occur between January and March 2023.

I. Consolidate the hosting plans

Once Lata drew up her "hosting empire", she added up the fees for these plans. She sorted them by fees incurred, starting from the highest to lowest. Most of her plans were paid annually or for a longer duration. Many were up for renewal over the next couple of months. This helped her to start prioritizing which hosting plans she should close. Terminating the most expensive plans alone could have added to her savings quite a bit, had it been a practical approach.

Estimated savings till March 2023 : about 40 US Dollars

Tip: List down the pricing for all the plans you have, that alone will be a great starting point before you start looking at savings.

II. Moving the hosting to a cheaper location(s)

Hosting in Europe typically used to be cheaper till recent months. Generally speaking, this is likely to remain for some time, depending on how the current energy crisis evolves. However, these times also present an opportunity to move the hosting to a different location - say North America. And, in some (very rare cases) also Asia.

Lata followed this approach and reached out to some of her hosting providers who had services across multiple regions. She asked them,

"Would they consider moving some of her services to cheaper locations?"

Some providers agreed, others refused, a couple of them offered to move her for a fee. Lata did a quick cost to benefit analysis, and decided that some non-critical service(s) could be moved to cheaper locations. However, the mission critical stuff that required the lowest latency or higher reliability remained at their present locations.

Estimated savings till March 2023 : about 35 US Dollars

Tip: Do you have hosting plans with provides who have services across multiple regions? Maybe a chat with the support team could help you figure out some savings.

III. Lower the specifications a notch or two

The third approach Lata adopted involved asking an all-important question:

Did she really needed the plans with higher specifications- hosting she had subscribed to "Just in case"? 

For example, she had a shared hosting plan in North America with about 20 GB of Storage space, with a maximum of 10 domains, and 1 TB bandwidth per month. Most of this capacity was lying unused. She decided to move to a lower plan that had half the specifications. This resulted in nearly 30 percent savings.

Estimated savings till March 2023: about 12 US Dollars

Tip: In times of tightening the belt, scaling down a plan or two can help save some costs.

The savings do add up, and the journey is not yet complete.

By adopting the above three approaches, (consolidating the hosting plans, moving the hosting plans to cheaper locations, and lowering the specifications on the plans) Lata is projected to save around 87 US Dollars over three months (or about 30 US dollars a month) in hosting fees. And she was not done yet! In the second part of this series, let us explore some more ways in which you can learn form Lata's experience and determine ways to lower your own web hosting costs.

Note: This article was originally written in October 2022, the estimated savings and timelines have been revised to reflect the costs at the time of publication.

Welcome to this first post of 2022 on the LES Blog!

Nearly 14 months have passed since the seminal announcement by Red Hat about the End of Life (EOL) for CentOs8. The much loved server oriented Linux distribution did reach it EOL on December 31, 2021. Over this period, how have the users coped up? What are the alternatives they have settled for? Did the predictions of doom and gloom really come true? Finally, which of the several alternative projects have made the greatest headway or found the highest user acceptance? Read on to learn more.



Source: Wikimedia Commons

You can listen to the audio version of this post at (External Link)

In This Post

1. Background: RedHat Discontinues development for CentOs8
2. Every Adversity is an Opportunity
3. Do We Have "Good" Alternatives?
4. Alma Linux and Rocky Linux
5. Other Alternatives to CentOs8
6. What About CentOs Stream and CentOs7?
7. Wrapping Up: End of CentOs8 Era
8. Optional reading - For Fun's Sake

Background: December 2020- CentOs project closing its doors

In early December 2020, the folks at Red Hat Linux had made the announcement that CentOS 8, the latest version of the open source server oriented operating system, would reach its end of life on December 31, 2021. Development of Centos 9 would stop. CentOs project would focus on something called CentOs Stream, instead of a server distribution that was downstream of Read Hat Linux.

Naturally, there was a lot of hue and cry among the users of CentOS in particular, and the open source community in general. The recurring view expressed was that an open source project that was backed by a large company, which decided to shut it down, leaving the users abandoned. The large company, Red Hat Linux, had itself been acquired by a much bigger tech company, - none other than Big Blue a.k.a IBM.

Every Adversity is an opportunity?

Since the day of the announcement, several expected and un-expected things happened.

a. There was an uproar in the tech community, and discussions on Reddit and web hosting forums, including this one.

b. Two new major projects were announced: Rocky Linux and Alma Linux. Both are backed by well known names in the Linux world. The trend of announcing new alternatives to CentOS continues. As recently as January 2022, SuSe Linux have proposed their own alternative to CentOs, called Liberty Linux. And if you scroll below to the "Piece De Resistance" section below, you may be in for a surprise.

c. Media houses, publications, bloggers and YouTubers all began to come up with "Alternatives to CentOs" type of posts. Some posted good alternatives, others simply drew up a random lists of operating systems which included Windows 10!

d. Competitors such as Oracle began to run promotions trying to woo away users into the fold of Oracle Linux.

They say, every adversity is an opportunity. Maybe for the real change makers over the long run. In reality, in the short term, it is the opportunists who can also benefit.

What turns out to be true in this case? Read on to find out...

It's good to have alternatives... but do we have "Good" alternatives?

My main intention here is to see where things stand nearly 14 months after the Red Hat announcement was made. In early February 2022, we will be in the fifth or sixth week since CentOs 8 reached its end of life. given this timeline, some of the questions I was keen to understand were:

a. Many alternatives to CentOs were proposed as in the multiple lists prepared by different publications. You can read some of them here and here. Have the users really migrated to one of the many alternatives that are that were proposed?

b. What made them choose one option over another? Did any move to CentOs Stream?

c. What is the feedback about these alternatives? Which one really gained traction among the users?

Taking a Look: User feedback and beyond

Alma Linux and Rocky Linux - The Dynamic Duo

While researching for this article, it has become evident that the two most popular CentOS alternatives are AlmaLinux and Rocky Linux. The former is created by the folks at Cloud Linux, while the latter is set up by one of the original co-founders of CentOS.

No matter which publication or resource once one looks at, these two names keep coming up right at the top.

Source: Alma Linux

Alma Linux has a very good comparison of features, organizational structure, etc. for some of the alternative If you scroll below, you will find a table comparing the different alternative: CentOs, Alma Linux, Rocky Linux, Oracle Linux and Red Hat . Alma Linux is an open a community based distribution from the folks at Cloud Linux. Rocky Linux is a private limited company started by one of the original co founders of CentOS. The former is a nonprofit, latter is a for profit setup. They both offer scripts that help the users migrate from CentOS to their respective distributions. Compatibility with Red Hat Enterprise Linux support, end of life cycles etc are pretty much identical.

Source: Alma Linux

Some of the limitations of the alternatives were discussed extensively across Reddit, HackerNews and other sources. Some of the below also came up in this discussion right here on LES. They include, in no particular order,

• Community, or lack of
• Longevity of the teams or support from the organizations behind the projects
• Maturity of the products
• Backing by large Corporates. Some people were not happy with Corporate support for some of the newer projects- the Red Hat experience probably left a bad taste.

Computingforgeeks has a good list of features that discusses the pro's and con's of the different alternatives to CentOs. You can also read this discussion on Reddit and on Hackernews. For those who prefer the Video version, here is a decent resource that explains the difference between Rocky and Alma Linux.

Update April 2022: Distrowatch.com in their April 4, 2022 edition of weekly newsletter (Issue 962) mentions an article by FOSS Force which talks about Alma Linux as the most favored replacement for CentOs.

The Next most discussed/ talked about alternatives to CentOs

Based on anecdotal experience and discussion with the LES community, I did not encounter many users who have taken up any of the other alternatives. This includes Oracle Linux. Granted this is a small subset of the user base, it does give a data point to be considered. With that in mind, I am listing a few alternatives that I have come across nonetheless.

Oracle Linux

Oracle Linux is based on Red Hat Enterprise Linux. Oracle even created a page on its website targeting CentOS users. They have developed a script that can convert existing CentOS installations to Oracle Linux. This distribution is free to use, but support contract is available for a charge.


Source:Wikimedia commons

According to the Wikipedia page for this distribution,

Oracle Linux is ..."available partially under the GNU General Public License since late 2006.[4] It is compiled from Red Hat Enterprise Linux (RHEL) source code, replacing Red Hat branding with Oracle's."

Liberty Linux (project by Suse Linux)

This is the newest kid on the block, I had posted about it in the LES forum. Too early to discuss or comment, pending further updates, I will leave you with the link to the LES discussion.

Springdale Linux

This is a project by the members of the computing staff of Princeton University and the Institute for Advanced Study, this was earlier known as PUIAS Linux. They describe themselves as a "Custom Red Hat based Distribution and Mirror." Among other things they have an Interesting Webpage !! The design reminds us of the early years of the World Wide Web.

visit Springdale Linux to learn more.

ClearOS

ClearOs offer commercial and community edition, as per the discussion on ClearOS forums, this may not be a long term viable alternative

VZLinux

This is a CentOs based distribution from folks at Virtuozzo. You can download the image files from:
https://repo.virtuozzo.com/vzlinux/8/iso/

Other than the above, I did not find much written or talked about this alternative to CentOs.

Fedora Server

Fedora is primarily a desktop oriented distribution, and a server version of Fedora, while possible and even capable, does not make a lot of sense. Atleast to me. According to one of the posts I came across on the topic,

Fedora server also follows short lifecycle. It is a community-supported operating system. Paid or commercial support option does not exist, and the lifecycle makes it less appealing as an alternative.

What about CentOs Stream?

Would you really consider moving to CentOS Stream for your servers? I have not found a definitive "Heck Yes" or "Hell, No!" answer to this query. Nor have, it appears these guys.



Source: Postgresweb

Are you perfectly happy with CentOs 7?

If you are happy with CentOs7, or belong to the if-it-ain't-broke-don't-fix-it school, then you have atleast 30 plus months of updates for CentOs coming from Red Hat. The Frequently Asked Question page on CentOs site confirms the same. In that case, should the EOL for CentOs 8creally be a concern for you?

And now, the pièce de résistance: A server Linux distribution from Microsoft

This one is real guys, and no more a joke in the Linux and open source communities. (Anybody remember Lindows.. er.. Linspire?)
edit: The project is still alive! That's cool.

Source: Wikipedia

The Linux distribution from Microsoft, called CBL-Mariner, is based off Fedora and Linux From Scratch. Form the github page for CBL-Mariner, https://github.com/microsoft/CBL-Mariner

CBL-Mariner is an internal Linux distribution for Microsoft’s cloud infrastructure and edge products and services. CBL-Mariner is designed to provide a consistent platform for these devices and services and will enhance Microsoft’s ability to stay current on Linux updates. This initiative is part of Microsoft’s increasing investment in a wide range of Linux technologies, such as SONiC, Azure Sphere OS and Windows Subsystem for Linux (WSL). CBL-Mariner is being shared publicly as part of Microsoft’s commitment to Open Source and to contribute back to the Linux community. CBL-Mariner does not change our approach or commitment to any existing third-party Linux distribution offerings.

Wrapping it up: Nearing the end of the CentOS Era

It is great to see several alternatives for CentOs, which is one of the beauties of the Open Source world. The challenge, as always, is that it presents a problem of plenty. Many folks in the LES Community- web hosts or the advanced users who used CentOs previously, seem to have converged towards Alma Linux or Rocky Linux. Among the rest of the list- Oracle Linux might have some users, given that many members on both the green forums have atleast one instance of Oracle Free tier.

I hope that this post has captured the "After the Storm" scenario since CentOS8 reached EOL. Over the long run, I intend it will offer a handy resource for the LES community. In addition to the top two alternatives (Rocky Linux and Alma Linux), there is a long tail of "also consider" alternatives. Which is one of the real beautiful things about the Linux and Open Source world.

Further (Optional) reading: FFs (For Fun's Sake)

While researching for this post, the search results threw some junk posts. I am adding a few of them below to add a flavor of humor. For example, really useless practices by some of the reviewers and authors. These included:

Some reviewers did a desktop review of Alma Linux, as in literally installing graphical interfaces. They spent a lot of time talking about how GNOME performed on this distribution versus KDE. One person even, did a screen capture of games being played on Alma Linux. Not sure if any server admins run GUI or play Steam games on their production servers.

Source: HowtoShout

The Techmint guys aren't too far behind : https://www.tecmint.com/wp-content/uploads/2021/06/AlmaLinux-Desktop.png

Windows 10 as an alternative to CentOS? Really??
g2 smokes some cheap stuff.

The Roll your eyeballs version
Simply copy and paste names of popular Linux distributions, and not really doing any favor to the reader. Hostingseekers, you need therapy

Author's Note:

I put together this post with the intention of creating a handy resource for readers here on the Low End Spirit blog. I am not a user of CentOS or any of its derivatives. Even though I began my Linux Journey with Red Hat 6 (Zoot), my poison of choice for many years has been Debian. Or, grudgingly, I use Ubuntu LTS where the Software as a Service (SaaS) vendors require me to use it.

This post presented an opportunity first and foremost to understand what is going on in the CentOS world, or the CentOS extended universe. (To borrow a phrase from the Marvel Universe). I have not done any installation or testing of my own for the purposes of this article.My interest was mainly in understanding what has been the market acceptance over the past few months. One hopes that has been served well.

For any questions or comments, write to me at contact+les@amarvyas.in

1- Have your vm already installed with one of the provided templates e.g: debian 9. ssh into your vm as root install the ipxe package:

# apt-get install ipxe
# update-grub

Get your ip, gateway from your vm info email, provider cp or on debian this way and store it as it will be needed later.

# cat /etc/network/interfaces
or
# ip r

# reboot

2- From your vnc or console, you should be able to see similar grub menu

after you press enter then quickly keep pressing CTRL+B ,or you can set to boot from CD only in you cp... then you should

enter your ip info you stored from the above step 1 like show in next image

if your vm supports ipv6 then it may try to configure it so you press enter like shown:

let it boot and should be good

I wrote this article when i got a HH vm and uploading iso is broken at time of this article.

References :
1- https://unix.stackexchange.com/questions/190865/is-it-possible-to-add-some-pxe-network-boot-option-to-grub
2- http://www.panticz.de/Install-GRUB-iPXE-netboot

There are couple of providers offerings NAT vps such as inceptionhosting , mrvm and webhorizon .

Thanks to webhorizon for sponsoring this guide and offered to bump my ram.

NAT VPSs are affordable and useful for practical small apps. In this guide we will setup a docker swarm cluster with a mix of single KVM manager and couple of OpenVZ7 NATs workers. For high availability it is recommend to have an odd number < 5 managers.

Minimum requirements:
- 1 KVM and 1 NAT nodes with at least 512MB Ram and a minimum of 5GB disk
- using IPv4 on all nodes.
- The host OS is Alpine Linux version 3.15 which is latest at time of writing this guide. Webhorizon already provides alpine as an os template on their NAT range.

Section 1: ( Host setup and applications install )

Host kernel settings:
- All hosts should have those kernel settings on ; For NATs if not enabled then open a ticket and nicely ask.

# cat /proc/sys/net/ipv4/ip_forward
1
# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

for KVM you can add that by

cat > /etc/sysctl.d/01-configs.conf <<EOF
# alpine
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
EOF

If your NAT provider does not include Alpine Linux template then one can install it using this script and my slightly modified version script.

On your NAT vps cp make sure you enabled

OpenVZ7 Install wireguard-go

# apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing wireguard-go
# apk add wireguard-tools

while the above is a testing repo so far it has been stable. You can always compile your own version.

KVM Install wireguard

# apk add wireguard-tools

Install and enable Docker

# apk add docker
# service enable docker

Section 2: Nodes details

1. Server "kvm"

   • Main Interface IP = 185.253.47.54
   • Wireguard Private IP = 192.168.1.2

2. Client "OpenVZ NAT"

   • Main Interface IP = 10.37.136.225
   • Wireguard Private IP = 192.168.1.10

Section 3: ( Generate keys and Conf file )

Wireguard requires key generation and setting up the network interface in a certain way. There are scripts and quick command lines to accomplish this task. The next details describe the manual method. Take a note of the cat outputs tp add in wg0.conf file.

generate keys those way on both nodes

# wg genkey | tee privatekey | wg pubkey > publickey
# cat publickey
     OaJ6KDPdgOs+aOJh7oEnaQYs/w9aIuHd0LfFgyx3ORw=
# cat privatekey
     WH8fN46zIMIXCxt2m5+I75vI+31oehh6aHte3qYYIys=
# # presharedkey is only needed for client node
#  wg genpsk > presharedkey
     Gg9HaCMkeEzUNl5zz9fUEzaDKIO2MNUKeG3D+ihchS8=

Server side config

# mkdir -p /etc/wireguard
# nano /etc/wireguard/wg0.conf
     [Interface]
     ListenPort = 51820
     PrivateKey = WH8fN46zIMIXCxt2m5+I75vI+31oehh6aHte3qYYIys=
     
     [Peer]
     PublicKey = OaJ6KDPdgOs+aOJh7oEnaQYs/w9aIuHd0LfFgyx3ORw=
     PresharedKey = Gg9HaCMkeEzUNl5zz9fUEzaDKIO2MNUKeG3D+ihchS8=
     AllowedIPs = 192.168.1.10/32

Client side config ; change port number to one in the range given to you, in my example ist 22520

# nano /etc/wireguard/wg0.conf
     [Interface]
     ListenPort = 22520
     PrivateKey = +KoN0pd30I6WERz2s9itHlIoikZBh1q2cbhB88UsnlI=
     
    [Peer]
    PublicKey = OaJ6KDPdgOs+aOJh7oEnaQYs/w9aIuHd0LfFgyx3ORw=
    PresharedKey = Gg9HaCMkeEzUNl5zz9fUEzaDKIO2MNUKeG3D+ihchS8=
    AllowedIPs = 0.0.0.0/0
    Endpoint = 185.253.47.54:51820
    PersistentKeepalive = 25

Section 4: ( Update network configuration )

Server side interface file

# nano /etc/network/interfaces
    ...
    auto wg0
    iface wg0 inet static
            address 192.168.1.2/24
            netmask 255.255.255.0
            pre-up ip link add dev wg0 type wireguard
            pre-up wg setconf wg0 /etc/wireguard/wg0.conf
            post-up ip route add 192.168.1.0/24 dev wg0; /sbin/iptables -A FORWARD -i wg0 -j ACCEPT;/sbin/iptables -A FORWARD -o wg0 -j ACCEPT;/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
            post-down ip link delete dev wg0
            post-down  /sbin/iptables -D FORWARD -i wg0 -j ACCEPT;/sbin/iptables -D FORWARD -o wg0 -j ACCEPT;/sbin/iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

Client side interface file

# nano /etc/network/interfaces
    ...
    auto wg0
    iface wg0 inet static
        address 192.168.1.10/24
        pre-up /usr/bin/wireguard-go wg0
        post-up /usr/bin/wg setconf wg0 /etc/wireguard/wg0.conf
        post-up /sbin/ip route add 192.168.1.0/24 dev wg0;
        post-down /sbin/ip link del wg0
        post-down rm -f /var/run/wireguard/wg0.sock

Section 5: ( Reboot and Test )

Do a reboot and start testing the connection for both server and client, yours will be different when changing the ips, ports and keys etc...

# ip a

# wg show

above test show connection is made and our wireguard nodes are reachable.

This is the first in a series of posts in the "Easy Mode" series

I thought I would write this up as a super easy basic guide to an almost no-effort method of migrating a KVM VPS from one host to another without having to worry about any minor changes in software stacks that could cause your backup and restore to fail or not restore as expected.

As a host one of the unfortunate realities is that sometimes it is just necessary to have migrations either due to end of life of an OS major version or impending hardware failure etc and on the rare occasion you need people to migrate their own server, this usually results in a 10% panic rate and a desperate cry for help.

With all that in mind I thought I would illustrate possibly one of the most simple server migration methods possible, I have done this based on KVM to KVM as that is likely to be the most common method however this method should also work for Xen HVM to Xen HVM or KVM to Xen HVM or Xen HVM to KVM and finally VMware to VMware should also work fine.

I can't cover the huge amount of different control panels and access methods out there but all that I have used either have access to an ISO mount or a rescue mode so it should be easy for you to use this guide as long as your host's control panel provides at least an ISO mount option or a rescue mode.

In this example, I have used a KVM VPS on a host that uses SolusVM and migrated it to a host that uses Virtualizor.

Requirements and additional info before we get started.

• The destination VPS should have the same size hard disk (bigger is fine).
• This will only go as fast as the link between the 2 locations, latency is a huge factor, moving a 50GB Disk image from India to New York is no fun.
• Some attention to detail is required, it is possible to lose your data if you mess up the commands.
• If the destination control panel does not provide a 'reconfigure networking' button or a serial console/VNC option you can use to reconfigure networking yourself you may want to consider configuring but not activating the network settings for the destination VPS on the source prior to shutdown, SolusVM, however, will do this for you and in some cases depending on the Hosts configuration options in Virtualizor it will also.
• Source VPS, in this case, was at Inception Hosting - London, the destination VPS was with Nexus Bytes - New York

First of all, I created a quick KVM VPS as the source on the SolusVM based host, installed apache2 and slightly modified the test page which you can see loading on the source IP:

Then I shut down the Source VPS and put both the source and destination VPS into rescue mode.

For SolusVM (Source):

For Virtualizor (Destination):

note: I noticed virtualizer has a tendency to just lock out all options even those for coming out of rescue mode when you do this which is resolved by simply refreshing the page, don't worry it does not resubmit the action

Now you should have both the source and destination in rescue mode and you should be able to login to both over ssh:

SolusVM VPS (Source)

Virtualizor VPS (Destination)

Next, you need to examine your disks to make sure you get the right one, the rescue modes will create a ramdisk so what was vda or sda on your VPS when it was running may not be now, this is the part you need to be sure you get correct, I have shown a side by side below using fdisk -l to check which disk I want to migrate to which disk on the other end.

From the image above we can see that the 5GiB disk is the one I want /dev/vda because I know my source disk is 5GiB and on the destination side I can see that it is actually /dev/vdb I want to copy the disk to, the 10 GiB disk. It is very important that you check this before going any further as it could well be different.

Because we now know I want to copy vda on the source to vdb on the destination we can structure the command to do this.

dd if=/dev/vda bs=32M status=progress | ssh root@45.61.123.123 "dd of=/dev/vdb"

let us break this down so it is better understood as there is nothing worse than knowing which buttons to press to achieve a goal, but having no idea why they work, or what they actually do:

• dd - Short for "data duplicator" a common commend used for copying or duplicating data.
• if=/dev/vda - The input or what we are reading, in this case, the entire virtual hard disk image
• bs=32M - BYTES, the number of BYTES to read per block when copying, you can set this yourself, I have found 32M is a good average.
• status=progress - This shows the progress of the copy, it has a very small overhead but without it you get nothing in terms of a progress indicator
• | - The pipe, in very simple terms, lets you pass the result of one command or set of commands to the next
• ssh root@45.61.123.123 - Because this is after the pipe | it takes what was done before and sends it over an ssh session as root to the destination IP 45.61.123.123 (fake)
• "dd of=/dev/vdb" - Finally the instruction on what to do at the other side which is to use dd "data duplicator" to write the blocks it is receiving over ssh to /dev/vdb

Once that command is issued you will see something like this:

And when it completes you will see:

As we can see that took 634 seconds or a little over 10 minutes to complete and now the disk image on the destination should be identical to the source.

The next step is to cancel rescue mode on the destination via the control panel and then boot the destination VPS, going to the new IP address once it is finished booting shows that it was a success as the same page loads on the new IP:

If your destination VPS has a bigger disk size than the source you will not instantly gain the extra space by following the above process, you will need to either expand your partitions or extend your logical volume if using LVM, that however is beyond the scope of this particular tutorial but if you are also looking for easy mode for that then just boot the VPS with a GParted ISO or a rescue cd that contains GParted which will give you a simple UI to expand your partitions or look up vgextend and lvextend if using LVM.

If having read this you discover that your host does not have a built-in rescue mode then I would suggest requesting they supply a SystemRescueCd ISO (Also contains GParted) https://www.system-rescue-cd.org/ so you can mount it and boot into rescue mode.

I. Before We Start

We need to obtain our basic network configuration from our provider. Or, if we are running our own host node, we need to assign basic network configuration to ourselves. Our basic network configuration might look something like this:

For IPv6, one might expect something like:

But occasionally, IPv6 could be something like:

Notes:

• The /28 in the IPv4 address and the longer netmask are different ways of providing the same information about the size of the local, directly connected network. It suffices for us to have this information in one format or the other. We don't need both formats because the information is the same. Also, the broadcast IP might not be provided, since it isn't strictly necessary.
• For the second format of the IPv6 address, what happened to the /64? 😱 The /128 in the second form of the IPv6 address might seem clueless to IPv6 fans expecting a /64. Also, the second format of the IPv6 address includes a gateway6 address. The gateway6 address might seem strange to some IPv6 fans, but we need the gateway6 for our minimal, static configuration. More on all this below.

II. Introduction

In the previous post of this series we finished using the Proxmox web GUI to install our new Debian KVM VPS via the Debian netinst installer iso image. The final step in Part II was removing the netinst install iso image from the emulated cdrom and then reooting our new VM, which came up from its own internal filesystem:

In today's post, we continue from this exact place where we left Part II -- connected to our newly installed and newly rebooted KVM via the Proxmox web GUI. In this post, we will accomplish the networking configuration which was skipped in Part II because the Debian netinst iso doesn't automatically configure out of band IP addresses.

There are three network configuration and related tasks we will accomplish today:

• First, we go "inside" our VM through the Proxmox web GUI's emulated "physical" console connection and set up networking. In Debian, networking setup requires that we adjust the file /etc/network/interfaces to tell our VM its network address and the address of its gateway to the internet.
• Second, we edit the file /etc/resolv.conf to tell our VM the numerical addresses of Domain Name System ("DNS") servers it can use to translate human readable Uniform Resource Identifiers (URI) into numerical Internet Protocaol ("IP") addresses.
• Third, we set up /etc/apt/sources.list to tell our system's Aptitude software package manager ("APT") where to get software updates and the additional software packages we will want to install.

Section III, Quick Setup, runs quickly through all three of today's tasks in "recipe style."

Section IV offers additional context on our setup environment.

Sections V, VI, and VII provide additional details on today's three setup tasks.

Section VIII discusses security.

Section IX discusses backup.

When we finish the Quick Setup, our new Debian KVM VPS should be connected to the internet, DNS should work, and we should be able to use the Debian package system to add whatever additional software we want.

When we finish all of today's post, we should have reasonable context within which to understand our Debian VM's networking setup.

III. Quick Setup

Logged into our VM through the Proxmox web GUI, we run the command ip link show. This command will give us the name of our network interface, probably something like "ens18."

As root or with sudo, we edit the text of the file /etc/network/interfaces so that it contains the minimum necessary information:

auto ens18
iface ens18 inet static
  address IPv4_ADDRESS/CIDR
  gateway GATEWAY_ADDRESS

iface ens18 inet6 static
  address IPv6_ADDRESS/CIDR
  gateway GATEWAY6_ADDRESS

Using our example network configuration, our minimal /etc/network/interfaces looks like this:

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
  gateway 172.16.164.1

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
  gateway fe80:xxxx:xxxx:xxxx::3

Second, we edit the /etc/resolv.conf file so that it looks like this:

nameserver 1.1.1.1
nameserver 8.8.8.8
nameserver 2606:4700:4700::1111
nameserver 2001:4860:4860::8888

Third, we edit /etc/apt/sources.list so that it looks like this:

deb http://deb.debian.org/debian buster main contrib non-free

deb http://deb.debian.org/debian-security/ buster/updates main contrib non-free

deb http://deb.debian.org/debian buster-updates main contrib non-free

Finally, we restart networking so that our new configuration takes effect:

systemctl restart networking

At this point, we should have both IPv4 and IPv6 connectivity, and DNS and APT both should work.

IV. More Context

• Virtualized Console Connection

The Proxmox web GUI virtualizes a wired console connection. In other words, our web browser does connect over the internet to our Proxmox server, but, the view from inside our new KVM is the same as though a wired connection was attached. Our new KVM thinks it's talking over a wired connection to a physical console. From inside our new KVM, there is, as yet, no network connection.

By default, the Proxmox web GUI works via VNC. In the Proxmox wiki on serial terminal Proxmox warns that VNC might

not have the features you need (i.e. easy copy/paste between other terminals)

or it might be

impossible to capture all [kernel messages, standard output, or error] messages on [the] VNC screen.

Yep, copy / paste commands do not seem to work in the Proxmox KVM virtual console.

Also, if you enjoy using the vi editor, you might find what looks like a "Send-Esc" button among the set of choices within the set exposed by the top button on the console VNC control bar. Use of the real keyboard Escape key results in exiting full screen. However, a second real Esc seems to produce the expected mode change, despite that maybe we no longer can see too well without returning to full screen.

• No DHCP, No SLAAC

These days most network setups use Dynamic Host Configuration Protocol (DHCP) to autoconfigure IPv4 networking. The machine on which networking is to be configured asks for and receives from a DHCP server all the needed information for the networking setup.

It is possible to configure DHCP so that it always returns the same IP address to each VM, but, since our entire Proxmox network is static, it may be simpler to set up networking manually--the traditional way for servers.

Stateless Address Autoconfiguration ("SLAAC") provides automatic configuration of IPv6 addresses. SLAAC requires a /64, which is why people say, for IPv6, that a /64 is expected and that less than a /64 is clueless. However, it remains possible to hand configure a single static IPv6 address, as we are doing here.

What if, for whatever reason, we simply do not want to use SLAAC? What if our provider doesn't receive enough IPv6 addresses from his provider to allow passing on to each VPS its own /64? What if our provider's provider charges an extra fee for extra IPv6 addresses, but we do not want to pay our provider's pass through of his provider's extra fee? What if we simply choose to use single, static IPs as is traditional for servers?

• No Cloud-Init

As mentioned in the previous post of this series, most VM network setups these days are done with Cloud-Init. Proxmox supports Cloud-Init, which enables both networking and ssh access to virtual machines to be set up on the Proxmox hypervisor and outside of the VM. Cloud-init can use DHCP. Here, however, we have chosen the simplest possible manual configuration with static IPs.

• Our Static, Routed Configuration And Out of Band Gateway From Our Provider's Provider

Here, our single, static IPv4 and single, static IPv6 are each derived from a routed subnet assigned to our server node. However, our internet gateway IPv4 address is not included among our server's routed group of IPv4s. This is called an "out of band" gateway.

Besides routed subnets, it also is possible for a datacenter to assign to servers non-routed, individual IP addresses. Data for these non-routed IPs moves between the datacenter switch and server nodes via the "link layer." Hetzner has a tutorial on Debian network configuration which includes discussion of "bridged configuration" for non-routed IPs.

• Systemd in Debian Networking

Since about 2014, networking is setup on Debian with systemd. The choice of systemd initially was and has continued to be divisive. Nevertheless systemd has remained as the Debian default.

There are at least two basic variations of Debian's systemd network arrangement. The first--which seems to be the default variation for Debian systemd network configuration--at least with the netinst iso--is using systemd's networking.service. For example, by using systemctl, we can confirm that networking.service is what is being used on our Node:

root@Proxmox-VE ~ # systemctl status networking.service
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: 
   Active: active (exited) since Wed 2021-06-02 19:13:13 UTC; 1 weeks 2 days ago
     Docs: man:interfaces(5)
 Main PID: 791 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/networking.service

 [ . . . ]
root@Proxmox-VE ~ # 

Our test KVM also seems to think its networking is controlled by systemd:

root@debian-kvm:~# systemctl status networking
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2021-06-16 01:20:45 UTC; 4min 51s ago
     Docs: man:interfaces(5)
  Process: 448 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
 Main PID: 448 (code=exited, status=0/SUCCESS)

Jun 16 01:20:45 debian-kvm systemd[1]: Starting Raise network interfaces...
Jun 16 01:20:45 debian-kvm systemd[1]: Started Raise network interfaces.
root@debian-kvm:~#

As we can see, systemd networking.service calls the traditional debian ifup and ifdown.

root@debian-kvm:~# cat /lib/systemd/system/networking.service
[Unit]
Description=Raise network interfaces
Documentation=man:interfaces(5)
DefaultDependencies=no
Requires=ifupdown-pre.service
Wants=network.target
After=local-fs.target network-pre.target apparmor.service systemd-sysctl.service systemd-modules-load.service ifupdown-pre.service
Before=network.target shutdown.target network-online.target
Conflicts=shutdown.target

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

[Service]
Type=oneshot
EnvironmentFile=-/etc/default/networking
ExecStart=/sbin/ifup -a --read-environment
ExecStop=/sbin/ifdown -a --read-environment --exclude=lo
RemainAfterExit=true
TimeoutStartSec=5min
root@debian-kvm:~# 

The second Debian systemd possibility--not the default on Debian netinst.iso and not used here--is systemd-networkd. Sahitya Maruvada has a simple, clear, Debian systemd-networkd introduction, Working with systemd-networkd. The systemd-networkd wiki page and the systemd.network manpage also are available.

• Official Debian Network Setup Instructions

Official Debian network setup instructions include the Wiki, the Handbook, manual pages such as man interfaces, /etc/network/interfaces examples online, and sometimes locally:

# less /usr/share/doc/ifupdown/examples/network-interfaces

• The ip Command Usually Is Available Even Though Networking Setup Varies Among Linux Distributions

Setting up networking, DNS name resolution, and software package management is very different in different Linux distributions. Therefore, we should not assume that the steps taken below would be exactly the same with a different Linux distribution than Debian.

Nevertheless, despite the different distributions' differing network setup systems, the ip command, supplied by the iproute2 collection, usually is available these days. Please see also Red Hat's IP Command Cheat Sheet

Because the ip command often is available, networking can be configured in many distributions, including Debian, by running a sequence of ip commands. The net effect of the sequence of ip commands can be to get the network functioning on most distributions without touching that individual distribution's network setup scheme.

Here's an example of the ip command used in the context of an iPXE boot. Note that the first command in the linked example requires knowledge of the name of the interface. We can list the names of the interfaces on our system by running the ip link show command.

One issue with using a sequence of ip commands is that the network setup fails to persist across reboots. However, we can place the ip command sequence inside a script which will be run automagically every time the server reboots. The sequence of ip commands in a script reminds us of the days before systemd, when scripts controlled all parts of the boot process including network setup.

Our KVM VPS's internal network configuration that we will be using below is similar to how LXC containers are configured in Proxmox. As will be seen below, Proxmox's LXC containers' network configuration adopts a variant of the "scripted ip command" approach, which also works inside Proxmox's KVM VPSes.

V. Our VM's Network Setup

• Interfaces

Our original /etc/network/interfaces file, the one installed by the netinst.iso, might look like this:

debian@debian-kvm:~$ cd /etc/network
debian@debian-kvm:/etc/network$ cat interfaces.original
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback
debian@debian-kvm:/etc/network$ 

Note that, in the default from the netinst.iso, /etc/network/interfaces.d is empty, so sourcing its files does nothing to the configuration.

debian@debian-kvm:/etc/network$ ls interfaces.d
debian@debian-kvm:/etc/network$ 

Now, let's edit /etc/network/interfaces to match our example network information from the above Before We Start section.

debian@debian-kvm:/etc/network$ cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
  gateway 172.16.164.1

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
  gateway fe80:xxxx:xxxx:xxxx::3

debian@debian-kvm:/etc/network$ 

The minimum required information does not include comments (lines beginning with #). Maybe we can make the rash and short-sighted assumption that we are not going to install anything which will want a file included from interfaces.d. The loopback interface might no longer be required (please see lines 17 and 18 in this file from Debian sources). Thus, for our example setup, the minimum /etc/network/interfaces might be:

debian@debian-kvm:/etc/network$ cat interfaces

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
  gateway 172.16.164.1

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
  gateway fe80:xxxx:xxxx:xxxx::3

debian@debian-kvm:/etc/network$ 

When configuring Debian LXC containers, Proxmox configures their /etc/network/interfaces files using added post-up and pre-down routes. Similarly, just for fun, instead of giving the gateway addresses in our /etc/network/interfaces,, we can manually add routes. Except for the initial post-up and pre-down these added lines mirror ip route commands that we could run manually to set up or take down networking without touching the /etc/network/interfaces file.

debian@debian-kvm:/etc/network$ cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
  address 172.16.165.97/28
     post-up ip route add 172.16.164.1 dev ens18
     post-up ip route add default via 172.16.164.1 dev ens18
     pre-down ip route del default via 172.16.164.1 dev ens18
     pre-down ip route del 172.16.164.1 dev ens18

iface ens18 inet6 static
  address fe80:xxxx:xxxx:xxxx::97/128
     post-up ip route add fe80:xxxx:xxxx:xxxx::3  dev ens18
     post-up ip route add default via fe80:xxxx:xxxx:xxxx::3  dev ens18
     pre-down ip route del default via fe80:xxxx:xxxx:xxxx::3  dev ens18
     pre-down ip route del fe80:xxxx:xxxx:xxxx::3  dev ens18

debian@debian-kvm:/etc/network$ 

VI. Our VM's DNS

We might want to add more or different nameservers to /etc/resolv.conf. Our Quick Setup configuration, above, includes IPs from Cloudflare and from Google.

VII. Our VM's Apt Setup

The Debian wiki instructions for configuring apt are at https://wiki.debian.org/SourcesList. There also is a man page. The configuration shown above, in Section III Quick Setup, is from the SourcesList Debian wiki page.

The Debian Security Information page says:

You can use apt to easily get the latest security updates. This requires a line such as
deb http://security.debian.org/debian-security buster/updates main contrib non-free

Many of the larger providers offer Debian mirrors. For example, Debian packages and security updates are available from the Hetzner Debian Mirror

After /etc/sources.list is edited, we update our system's package repositories as follows:

apt-get upgrade && apt-get dist-upgrade -y

We can see exactly which packages are installed by looking at the logs in /var/log/apt.

We may wish to install openssh-server so that we can connect to our VM via ssh in addition to our Proxmox VNC connection. With ssh we regain cut and paste functionality while enjoying lower apparent latency!

apt-get install openssh-server

The Kennedy article, mentioned below in Section VII, has some good tips for ssh server configuration.

VIII. Security

Google suggests its first choice among essential server security articles. This article from 2013, by Bryan Kennedy, seems to provide still-good advice, except that, nowadays, many people prefer to use ed25519 keys

IX. Backup

After all this work, we certainly want to make an offline backup of our new VM. We can use Proxmox to make the backup and then download a a copy from the host node's /var/lib/vz/dump directory.

This is a guest post by forum user @alexxgg,

Hi there!

These days when the internet shares things in milliseconds, who of you have needed to share a file quickly? I bet some of you remember at least one time one of those cases.

As you probably know, in terms of web servers, we can share files with popular software like Apache, Nginx, and Lighttpd but this software need basic configuration, also they consume server resources as long as they’re active.

What if you could set up a basic HTTP web server without installing Apache, Nginx, or Lighttpd? Well, that’s sounds kind of impossible, and even more unbelievable is that you can kill it with Ctrl^C. Thanks to Python3 we can do that with its HTTP server module.

This module will deploy an HTTP server in any directory of the server, even in the root directory and that sounds ridiculously dangerous, fortunately, the default port of this HTTP server isn’t 80. Instead, it will use port: 8000 but you can assign a custom port, very convenient for NAT environment instances.

Now you can deploy a basic HTTP server -in the current directory with the default port- by typing this single command line:

python3 -m http.server

Starting HTTP Server

Killing HTTP Server

HTTP Server as Shown in Browser

You can change the default port (example port: 32085) and specify a directory (example directory: /tmp/) with:

python3 -m http.sever 32085 --directory /tmp/

More information about Python3 httpserver module is available here

Also, there is a github page here

Note: of course, you will need to install Python3 in order to use these post example command lines.
Leave a comment to let me know any questions or suggestions.

This is a really QUICK TIP!

When you run any command as root, using sudo, the password is remembered for 15 minutes by default.
If you want to change the time that the password is cached, open the terminal (as root) and run:

editor /etc/sudoers 

Find this line in the file:

defaults env_reset

And change it into:

defaults env_reset , timestamp_timeout=x

where “x” is the time in minutes that the password will be cached.

Save and exit and work is done!

Cloudflare has recently adapted their SSL for SaaS product, which was originally limited only to their Enterprise customers but has now been released publicly. Yesterday they announced their Cloudflare for SaaS solution available to everyone. And to reflect their recent service evolution of the product they have re-named it to: Cloudflare for SaaS.

You may ask yourself: What exactly is SaaS?
Software as a Service (SaaS) is a method of providing software to an end-user where the software is not installed and maintained by the user, but via hosted services (most often through a web browser). Popular examples of such a service include Salesforce, Google Apps, Microsoft Office 365, etc.

Since the 15th of April, they have released their beta stage, which is available for sign up if you submit your request here. Before the introduction of Cloudflare for SaaS, your best bet was for your customer to set up a CNAME record and have them generate a private key and CSR.

On top of that, you would be required to maintain a solution to generate and securely store private keys. But with the introduction of Cloudflare for SaaS, it is now more freely accessible and has provided less of a burden when managing multiple customers.

Cloudflare Workers

Back in January when SSL for SaaS was announced, 80% of beta users were already building their application on Cloudflare workers. But by combining the use of Cloudflare for SaaS with Cloudflare workers, will overall reduce the time and core resources from building your application.

Security

It also has already provided DDoS Protection and Web Application Firewall (WAF) built-in. Alongside the benefits offered to the Enterprise customers at no additional cost.

So, if you want to have more control of your SSL certificate solution and would like a simple but manageable solution, then why not sign up for the beta.