LES BSD Thread!

linveo · September 21

@Crab said:
I noticed Linveo VirtFusion panel had changed its appearance, so I was wondering if a newer version was installed with potentially better support, but booting NetBSD and OpenBSD ISOs failed the same old way.

We did upgrade to VF 5.0 which brings a lot of changes. One of them to note for the FreeBSD crowd:

Added support for FreeBSD memory usage display.

There are some behind the scenes setting changes, so I am taking a look to see if there is anything to help with the current issues.

AlwaysSkint · September 21

^ I prefer the new layout, if nothing else.

Crab · September 21

@linveo said:

Added support for FreeBSD memory usage display.

I tried this out since I was curious about its implementation. By default it is trying to access /proc/meminfo which works fine in Linux but doesn't exist in *BSD out of the box. It actually uses freecolor for the memory information on FreeBSD and the VF panel shows the memory information just fine with it.

So people who might've done an ISO install, you'll need to install qemu-guest-agent and freecolor from packages to make it work.

cmeerw · September 21

I think I managed to extract the NetBSD kernel panic now. This is possible by using grub2 to read the memory where NetBSD formats the panic string. Essentially, what you need to do is look at either the netbsd.map file or the output of nm netbsd and look for the address of panicstr (but only use the lower 28 bits of that address - the memory contents there point to the address of the actual string, again only use the lower 28 bits of that address).

So, I boot a NetBSD kernel in grub2 via knetbsd, it immediately reboots and in grub I then do:

dump 0x12ea280 8
a0 a2 2e 81 ff ff ff ff
dump 0x12ea2a0 384

This then gives me the hex dump of the panic message, which in my case is:

kernel diagnostic assertion "lp_max >= core_max" failed: file "../../../../arch/x86/x86/cpu__topology.c", line 163

Guess I now just need to figure out what goes wrong there...

Not_Oles · September 21

Congrats @cmeerw! Thank you so much for posting!

AlwaysSkint · September 21

@Not_Oles said: Is anyone else interested in trying this cool NetBSD install method?

Goes off to break something..

AuroraZero · September 21

cmeerw · September 22

@cmeerw said:
This then gives me the hex dump of the panic message, which in my case is:

kernel diagnostic assertion "lp_max >= core_max" failed: file "../../../../arch/x86/x86/cpu__topology.c", line 163

Guess I now just need to figure out what goes wrong there...

The "obvious fix" of just adding:

lp_max = core_max;

seems to work and I have now managed to install NetBSD 10.0 with such a modified kernel.

But it looks like something is not being reported correctly (by the CPU/virtualised CPU) here.

Crab · September 22

Excellent discovery @cmeerw !

Looking at the kernel code at https://cdn.netbsd.org/pub/NetBSD/NetBSD-current/src/sys/arch/x86/x86/cpu_topology.c lp_max means the maximum logical processors and core_max obviously maximum cores per package that it detects. If the former smaller value than the latter, kassert will issue a kernel panic.

FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)

FreeBSD detects 2 packages and 1 core which should be correct, but for NetBSD seems to detect it other way around.

cmeerw · September 22

Running cpuid in a Linux VM on that machine I get:

$ cpuid -1l1 | fgrep hyper-threading
hyper-threading / multi-core supported = false

So lp_max = 1

$ cpuid -1l4 | fgrep "in pkg"
maximum IDs for cores in pkg = 0xf (15)

and core_max = 16

AlwaysSkint · September 22

Somehow, after a few hiccups, I got NetBSD to boot and install, via Linux (Debian 11). This was on an idle Virmach VPS with plenty of resources - now to try the same on the Linveo one..

Not_Oles · September 22

@AlwaysSkint said: Somehow, after a few hiccups, I got NetBSD to boot and install, via Linux (Debian 11).

Congrats on getting NetBSD up and running!

Wanna give us a hint, please, about the method you used to install NetBSD?

Maybe a link to a how to?

Maybe a sentence or two about the hiccups and what fixes you applied?

Thanks so much! Best wishes!

AlwaysSkint · September 22

@Not_Oles said: W_a_nnna give us a hint

I'm kinda pee'd off 'cos after trying numerous failures, I didn't have the exact steps that finally got this to work, on the Virmach VPS. So now, I'm struggling to recreate it on the Linveo one. Primarily due to me making changes directly in grub boot, editing the default (Debian) boot, then hitting F10.
It went along the lines of the following, though here, showing an addition to /etc/default/grub..

  menuentry 'NetBSD' {
      load_video
      insmod gzio
      insmod part_gpt
      insmod part_bsd
      insmod xfs
      insmod zfs
      search --no-floppy --fs-uuid --set=root ce1e50b7-51ad-4707-a483-8003a82xxxxx
      echo    'Loading NetBSD ...'
      knetbsd /ramdisk.fs
  }

The uuid was taken from the existing Debian entries. The ramdisk.fs is intended to be a RAM-based installation kernel - though I wish I'd paid more attention to the exact one that worked.
My browser tabs include:
https://www.gnu.org/software/grub/manual/grub/grub.html#NetBSD
https://mirrors.mit.edu/NetBSD/NetBSD-10.0/amd64/installation/ramdisk/
http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/202409211400Z/amd64/binary/kernel/

I'll see if I can glean some clues from the Virmach instance eg. bootlog

It was a simple process, at least on the Virmach VPS and didn't need any fakeroot or similar. The network didn't autoselect at first but I remained in the installation TUI and (re-)chose network setup, which did pick up the virtual NIC and subsequent DHCP.
Gawd, the benefit of hindsight.

Not_Oles · September 22

@AlwaysSkint

Thanks for the correction!

sed 's/nnn/ann'

Thanks for the install info! More interesting stuff for me to try!

AlwaysSkint · September 22

Haven't found much more, though in /var/log/messages:

NetBSD 10.0 (INSTALL) #0: Thu Mar 28 08:33:33 UTC 2024
[email protected]:/usr/src/sys/arch/amd64/compile/INSTALL

Whereas post install/reboot:

NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC

AlwaysSkint · September 22

..Much later. I created a new Debian 11 VM on my Proxmox server and booted it up.
'history'

su - root <--(if not already logged in as root)
vi /etc/default/grub <-- (change grub timeout to say, 20)
vi /etc/grub.d/40_custom

!/bin/sh

exec tail -n +3 $0

This file provides an easy way to add custom menu entries. Simply type the

menu entries you want to add after this comment. Be careful not to change

the 'exec tail' line above.

menuentry 'NetBSD' {
load_video
insmod gzio
insmod part_gpt
insmod part_bsd
insmod xfs
insmod zfs
search --no-floppy --fs-uuid --set=root 2eb84507-bd74-432f-8bb3-c695f9xxxxxxx
echo 'Loading NetBSD ...'
knetbsd /netbsd-INSTALL.gz
}

As before UUID was copied from the default /etc/boot/grub.cfg entry.

cd /
wget http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/202409211400Z/amd64/binary/kernel/netbsd-INSTALL.gz
update-grub
reboot and display a console.

AlwaysSkint · September 22

Goes off to try the above with VirtFusion (again)..

.. retried with Linveo and the installation kernel doesn't startup (looping back to grub). So it looks as though there's a fundamental difference (perhaps a setting like pass-through) in Virtfusion, compared to both SolusVM and Proxmox hypervisors.

cmeerw · September 23

@AlwaysSkint said:
.. retried with Linveo and the installation kernel doesn't startup (looping back to grub). So it looks as though there's a fundamental difference (perhaps a setting like pass-through) in Virtfusion, compared to both SolusVM and Proxmox hypervisors.

That's what I have been talking about here all the time: https://lowendspirit.com/discussion/comment/186288/#Comment_186288 and https://lowendspirit.com/discussion/comment/186301/#Comment_186301

You'll need to compile the kernel with the change mentioned and use that kernel during installation.

AlwaysSkint · September 23

@cmeerw said: That's what I have been talking about here all the time.

Apologies but it was a little bit too cryptic for my advancing years. Now that I re-read your posts, I can see how it fits in. I know, it has taken me a while to reach the same point, though in my defense, it was from a position of no experience with BSD. Thanks for your efforts, of course, and I'll perhaps give your fix a try - setting up for a custom kernel takes quite a bit of time/effort. It may be interesting/useful if @linveo divulged the VF settings used when defining a VM, as a fix at source strikes me as the way to go.
Has anyone got a VM that uses Virtfusion, at a different provider?

[Edit:typos - Grr, laptop keyboard]

cmeerw · September 23

@AlwaysSkint said:

@cmeerw said: That's what I have been talking about here all the time.

Apologies but it was a little bit too cryptic for my advancing years. Now that I re-read your posts, I can see how it fits in. I know, it has taken me a while to reach the same point, though in my defense, it was from a position of no experience with BSD. Thanks for your efforts, of course, and I'll perhaps give your fix a try - setting up for a custom kernel take quite a bit of time/effort. It may be interesting/useful if @linveo divulged the VF settings used when defining a VM, as a fix at source strikes meas the way to
go.
Has anyone got a VM that uses Virtfusion, at a different provider?

BTW, as far as I understand the code, it also only affects Intel CPUs (AMD should be fine).

Ideally, NetBSD should be able to handle this case more gracefully, so I have brought it up on their tech-kern mailing list yesterday.

AlwaysSkint · September 23

@cmeerw said: it also only affects Intel CPUs

Makes sense that the two working VM that I have are both on (different) AMD CPUs.

AlwaysSkint · September 23

@cmeerw said: I have now managed to install NetBSD 10.0 with such a modified kernel

Can your kernel be made available?

cmeerw · September 23

@AlwaysSkint said:

@cmeerw said: I have now managed to install NetBSD 10.0 with such a modified kernel

Can your kernel be made available?

Yes: https://edge.cmeerw.net/netbsd-10.0-vm.gz

BTW, a relatively easy way to then install NetBSD is to put that (gunzipped) kernel file into a (relatively small) separate partition on the VM (not sure which filesystems the NetBSD loader supports - I had put it into a FFS partition). You can then use the installer image https://cdn.netbsd.org/pub/NetBSD/NetBSD-10.0/amd64/installation/cdrom/boot.iso and in the NetBSD loader boot the patched kernel instead (but then select the installer CD as the root device). Once the installation is complete, you again need to tell the NetBSD loader to use the patched kernel again. Then just copy the patched kernel into /netbsd (and you can then re-use the separate kernel partition as swap)

Not_Oles · September 23

@cmeerw said: Ideally, NetBSD should be able to handle this case more gracefully, so I have brought it up on their tech-kern mailing list yesterday.

Thanks @cmeerw!

Link for the curious:

https://mail-index.netbsd.org/tech-kern/2024/09/22/msg029737.html

AlwaysSkint · September 23

@cmeerw said: ..a relatively easy way..

Crikey! Glad that you didn't tell us a more difficult way!

linveo · September 23

It seems like there might be more success with AMD CPUs? If anyone wants to change their VM type from Intel to AMD let me know and I can switch it for you. I appreciate all of the feedback I've received so far.

AlwaysSkint · September 23

@linveo said: If anyone wants to change their VM type from Intel to AMD let me know..

I'll go for it; makes no odds to me. :-)

Not_Oles · September 23

@cmeerw said: BTW, as far as I understand the code, it also only affects Intel CPUs (AMD should be fine).

@linveo said: If anyone wants to change their VM type from Intel to AMD let me know and I can switch it for you.

@linveo Thanks for offering migration! Assuming that the issue at hand would affect any method of NetBSD install under Virtfusion (because Virtfusion, or Qemu under Virtfusion, apparently doesn't report or doesn't correctly report data that the NetBSD kernel needs) I also am going to say yes to migrating to AMD.

Please simply wipe my existing VM without further notice and please send me login info for the AMD VM. Or whatever else works for you. Also, no rush.

If Phoenix is okay for AMD, then I'd love to stay in Phoenix, which offers the lowest latency for me. But any other location is equally great!

Thanks again! Kindest regards!

AlwaysSkint · September 23

^ I've powered down and willing to start from scratch too. Given location is across The Pond, then anywhere will do ( though CA is never good due to the prominence of Asian traffic).

@Not_Oles said: I'd love to stay in Phoenix..

I would've too had I found a replacement job, post redundancy. Heat City, Arizona!

linveo · September 24

@AlwaysSkint said:

@linveo said: If anyone wants to change their VM type from Intel to AMD let me know..

I'll go for it; makes no odds to me. :-)

@Not_Oles said:

@cmeerw said: BTW, as far as I understand the code, it also only affects Intel CPUs (AMD should be fine).

@linveo said: If anyone wants to change their VM type from Intel to AMD let me know and I can switch it for you.

@linveo Thanks for offering migration! Assuming that the issue at hand would affect any method of NetBSD install under Virtfusion (because Virtfusion, or Qemu under Virtfusion, apparently doesn't report or doesn't correctly report data that the NetBSD kernel needs) I also am going to say yes to migrating to AMD.

Please simply wipe my existing VM without further notice and please send me login info for the AMD VM. Or whatever else works for you. Also, no rush.

If Phoenix is okay for AMD, then I'd love to stay in Phoenix, which offers the lowest latency for me. But any other location is equally great!

Thanks again! Kindest regards!

There are AMD nodes in all the DCs, so I have migrated your VMs over to them as they were with the same IP. Hopefully this ends up being more fruitful.

LES BSD Thread!

Comments

!/bin/sh

This file provides an easy way to add custom menu entries. Simply type the

menu entries you want to add after this comment. Be careful not to change

the 'exec tail' line above.