LES BSD Thread!

Not_Oles · December 13

Moving to Debian got the bump to 100G.

root@linveo:~# date
Fri Dec 13 02:43:16 AM GMT 2024
root@linveo:~# cat /etc/debian_version 
12.8
root@linveo:~# df -h 
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           389M  572K  388M   1% /run
/dev/vda3       100G  3.6G   97G   4% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/vda2       121M  142K  120M   1% /boot/efi
tmpfs           389M     0  389M   0% /run/user/0
root@linveo:~#

Duh! If I restore from the NetBSD backup, is that going to take the file space back to 50G? In which case I would have to start again at the beginning with @cmeerw's image? We will see. Maybe.

Crab · December 13

@Not_Oles said:
Duh! If I restore from the NetBSD backup, is that going to take the file space back to 50G? In which case I would have to start again at the beginning with @cmeerw's image? We will see. Maybe.

man resize_ffs

Not_Oles · December 13

@Not_Oles said:
Duh! If I restore from the NetBSD backup, is that going to take the file space back to 50G?

Yes.

In which case I would have to start again at the beginning with @cmeerw's image? We will see. Maybe.

@Crab said: man resize_ffs

Thanks @Crab! Will check the resize_ffs man page.

For now, I am back in to the NetBSD restored from the backup.

linveo# df 
Filesystem      1K-blocks         Used        Avail %Cap Mounted on
/dev/dk2         50313000     44380778      3416572  92% /
kernfs                  1            1            0 100% /kern
ptyfs                   1            1            0 100% /dev/pts
procfs                  4            4            0 100% /proc
tmpfs             1048440            4      1048436   0% /tmp
tmpfs             1048440            4      1048436   0% /var/shm
linveo# cat /etc/fstab
# NetBSD /etc/fstab
# See /usr/share/examples/fstab/ for more examples.
NAME=netbsd-root        /       ffs     rw,log  1 1
NAME=netbsd-swap        none    swap    sw      0 0
kernfs          /kern           kernfs  rw
ptyfs           /dev/pts        ptyfs   rw
procfs          /proc           procfs  rw
/dev/cd0a       /cdrom          cd9660  ro,noauto
tmpfs           /tmp            tmpfs   rw,-m1777,-sram%25
tmpfs           /var/shm        tmpfs   rw,-m1777,-sram%25
linveo# man resize_ffs
  [ . . . ]
EXAMPLES
           resize_ffs /dev/vg00/rlv1

     will enlarge the file system on the Logical Volume /dev/vg00/lv1 from
     Volume Group vg00 to the current device size.
  [ . . . ]
linveo# resize_ffs /dev/dk2
It's required to manually run fsck on file system before you can resize it

 Did you run fsck on your disk (Yes/No) ? No

 Nothing done 
linveo# man fsck
 [ . . . ]
linveo# fsck /dev/dk2
fsck: cannot open `/dev/dk2': Device busy
linveo#

From reading the resize_ffs man page, it seems like it's not really "required" to run fsck. Instead maybe I could just use resize_ffs' -y option?

But, to run fsck, I might have to unmount /dev/dk2? If I unmount /dev/dk2, how do I retain access to the fsck binary which lives on /dev/dk2 in /sbin? Is fsck static? Do I copy the fsck binary to /tmp?

Hmm. There's an fsck which lives in /rescue. I never used the NetBSD rescue system before.

I have to look around more and read more. Additional hints are welcome! Thanks @Crab!

Crab · December 13

Can you enable VNC, reboot and add '-a' boot flag and put '/rescue/init' as the init path. All the tools at /rescue are statically linked.

Also https://www.netbsd.org/docs/misc/index.html#using-fsck

Not_Oles · December 13

@Crab said:
Can you enable VNC, reboot and add '-a' boot flag and put '/rescue/init' as the init path. All the tools at /rescue are statically linked.

Also https://www.netbsd.org/docs/misc/index.html#using-fsck

Thanks! I will try to try it tomorrow. Sleep now. Zzzzz!

@Crab You're a nice guy! Thanks again!

cmeerw · December 13

@Not_Oles said:

@Crab said:
Can you enable VNC, reboot and add '-a' boot flag and put '/rescue/init' as the init path. All the tools at /rescue are statically linked.

Also https://www.netbsd.org/docs/misc/index.html#using-fsck

Thanks! I will try to try it tomorrow. Sleep now. Zzzzz!

@Crab You're a nice guy! Thanks again!

So the main issue is that resize_ffs doesn't really handle the filesystem log, so resizing such a filesystem is disabled in my boot scripts. In some cases you could run into data corruption (although that would be with an out-of-filesystem log)

The clean way to do it is to clear the log (see tunefs -l 0 - which actually means to set a flag to clear the log, and mount the filesystem that then clears the log), resize the filesystem, and the log will then be recreated on the next mount. It's just painful to do that on the root filesystem (as that can't be completely unmounted).

Alternatively, you could just disable the check in /etc/rc.d/resize_root and let it resize the filesystem anyway (I think, it will just mean that the log will not be resized)

cmeerw · December 13

@Not_Oles said: Thanks! I will try to try it tomorrow. Sleep now. Zzzzz!

This should work:

reboot into single user mode
tunefs -l 0 /dev/dk2
mount -ruo reload /dev/dk2 /
reboot (this should then resize the filesystem, and trigger another reboot)

in between these commands you can check with dumpfs -s / the wapbl lines (tunefs should set the flags to 0x2), and after the mount with reload, wapbl version should be 0x0 and the wapbl loc entries should all be 0 again.

cmeerw · December 13

@cmeerw said:

@Not_Oles said: Thanks! I will try to try it tomorrow. Sleep now. Zzzzz!

This should work:

reboot into single user mode

tunefs -l 0 /dev/dk2

mount -ruo reload /dev/dk2 /

reboot (this should then resize the filesystem, and trigger another reboot)

in between these commands you can check with dumpfs -s / the wapbl lines (tunefs should set the flags to 0x2), and after the mount with reload, wapbl version should be 0x0 and the wapbl loc entries should all be 0 again.

And now that I have figured out the exact commands of how to do it, I could probably even integrate that into the resize_root script.

Not_Oles · December 13

Hi @cmeerw!

Thanks for the suggested steps!

I tried them. Everything seemed to go as expected. However, a message flashed by during the first reboot. It said something like, "Not resizing logging not enabled." Then the second reboot went ahead, and succeeded, but the disk size seemed not to have increased. Do we need to enable logging?

Remember that I might have made a mistake somewhere. It's not necessarily that your steps are wrong. For example, I chose /bin/sh for the shell. Should I have chosen /rescue/sh?

More ideas, please? Thank you again so much!

Tom

AlwaysSkint · December 14

fdisk -l ?
Has the partition been extended?
Just a couple of thoughts.

cmeerw · December 14

@Not_Oles those kernel messages don't look like they should be expected (maybe that's a change with NetBSD HEAD?)

BTW, Not resizing $rootmp: logging unsupported should be the message you are seeing (and the tunefs, re-mount was supposed to disable logging while resizing)

cmeerw · December 14

@cmeerw said:
@Not_Oles those kernel messages don't look like they should be expected (maybe that's a change with NetBSD HEAD?)

BTW, Not resizing $rootmp: logging unsupported should be the message you are seeing (and the tunefs, re-mount was supposed to disable logging while resizing)

Ok... while it seemed to work with NetBSD 10.0 for me, it actually left the old journal unconnected on the filesystem, so that's not too good. You might want to do a fsck_ffs -f /dev/dk2 in single user mode now.

cmeerw · December 14

@cmeerw said:

@cmeerw said:
@Not_Oles those kernel messages don't look like they should be expected (maybe that's a change with NetBSD HEAD?)

BTW, Not resizing $rootmp: logging unsupported should be the message you are seeing (and the tunefs, re-mount was supposed to disable logging while resizing)

Ok... while it seemed to work with NetBSD 10.0 for me, it actually left the old journal unconnected on the filesystem, so that's not too good. You might want to do a fsck_ffs -f /dev/dk2 in single user mode now.

Alternative approach:

reboot into single user mode
tunefs -l 0 /dev/dk2
reboot, again into single user mode
mount -u /dev/dk2 /
mount -ru /dev/dk2 /
exit

Not_Oles · December 14

@cmeerw said:
Not resizing $rootmp: logging unsupported should be the message you are seeing

That looks like what I saw. But I am not sure because it scrolled by so quickly. Are messages like this one recorded somewhere?

@cmeerw said: left the old journal unconnected on the filesystem

May I please ask how you figured this out? Thanks!

@cmeerw said: You might want to do a fsck_ffs -f /dev/dk2 in single user mode now.

Will do!

@cmeerw said:

Alternative approach:

reboot into single user mode
tunefs -l 0 /dev/dk2
reboot, again into single user mode
mount -u /dev/dk2 /
mount -ru /dev/dk2 /
exit

Will try.

Thanks so much for helping so much!

Thanks again to Linveo for the nice VPS!

cmeerw · December 14

@Not_Oles said:

@cmeerw said:
Not resizing $rootmp: logging unsupported should be the message you are seeing

That looks like what I saw. But I am not sure because it scrolled by so quickly. Are messages like this one recorded somewhere?

Some of the output from startup scripts might end up in /var/log/messages, but I don't think this one does (that particular script uses # KEYWORD: interactive, so the output doesn't get redirected to the syslog processor).

@cmeerw said: left the old journal unconnected on the filesystem

May I please ask how you figured this out? Thanks!

The kernel messages from HEAD don't look very nice, so I thought I'd just do a fsck in case, and it added those journals to lost+found (so the mount with reload must have managed to just set the information in the superblock to 0, but then didn't clear the inodes as the filesystem was supposed to be only mounted read-only).

Not_Oles · December 14

I had some fun looking at

linveo# pwd
/etc/rc.d
linveo# cat -n resize_root 
  [ . . . ]
    68                  for opt in $(split_options "${fs_mntops}"); do
    69                          if [ "$opt" = "log" ];  then
    70                                  echo "Not resizing $rootmp: logging unsupported"
    71                                  return
    72                          fi  [ . . . ]

I don't think I understood much of the sh code, but it was fun to be able to find quickly and easily what might seem to be the code where the message we are discussing originated.

linveo# pwd
/usr/src/sbin/resize_ffs
linveo# ls
CVS           Makefile      TODO          resize_ffs.8  resize_ffs.c
linveo#

I don't think I understood much of the C code, but I really appreciated the helpful comments in resize_ffs.c. It seems like mouse really wants people to be able to understand his code!

Thanks to @cmeerw, @linveo, and the NetBSD developers! And LES!

Not_Oles · December 14

@cmeerw said:

# KEYWORD: interactive

Okay, I see this in line 8. Thanks again!

linveo# pwd
/etc/rc.d
linveo# cat -n resize_root 
     1  #!/bin/sh
     2  #
     3  # $NetBSD: resize_root,v 1.6 2023/10/04 00:04:42 gutteridge Exp $
     4  #
     5
     6  # PROVIDE: resize_root
     7  # REQUIRE: fsck_root
     8  # KEYWORD: interactive
  [ . . . ]

cmeerw · December 14

@Not_Oles said:
I had some fun looking at

linveo# pwd
/etc/rc.d
linveo# cat -n resize_root 
  [ . . . ]
    68                  for opt in $(split_options "${fs_mntops}"); do
    69                          if [ "$opt" = "log" ];  then
    70                                  echo "Not resizing $rootmp: logging unsupported"
    71                                  return
    72                          fi  [ . . . ]

Oh... that's not good. That version of the script is a bit older than I thought (I though you already had the version that did the check via dumpfs instead of the flag in /etc/fstab)

That means you'll have to do the resize command in the shell as well (instead of relying on the resize_root script).

So I think it would then be:

reboot into single user mode
tunefs -l 0 /dev/dk2
reboot, again into single user mode
mount -u /dev/dk2 /
mount -ru /dev/dk2 /
resize_ffs /dev/dk2
reboot -n

cmeerw · December 14

@cmeerw said:
@Not_Oles said:
I had some fun looking at
linveo# pwd
/etc/rc.d
linveo# cat -n resize_root 
  [ . . . ]
    68                  for opt in $(split_options "${fs_mntops}"); do
    69                          if [ "$opt" = "log" ];  then
    70                                  echo "Not resizing $rootmp: logging unsupported"
    71                                  return
    72                          fi  [ . . . ]
Oh... that's not good. That version of the script is a bit older than I thought (I though you already had the version that did the check via dumpfs instead of the flag in /etc/fstab)

That means you'll have to do the resize command in the shell as well (instead of relying on the resize_root script).

So I think it would then be:

reboot into single user mode

tunefs -l 0 /dev/dk2

reboot, again into single user mode

mount -u /dev/dk2 /

mount -ru /dev/dk2 /

resize_ffs /dev/dk2

reboot -n

Ahh no, you probably got that resize_root script from NetBSD HEAD when you updated NetBSD.

Not_Oles · December 14

@cmeerw said:
Ahh no, you probably got that resize_root script from NetBSD HEAD when you updated NetBSD.

@cmeerw I guess you mean that there is a newer version of the script in your NetBSD 10 image than the version of the script which is in NetBSD HEAD?

I guess you mean that the version of the script which is in your NetBSD 10 image is the same as in the latest NetBSD 10 on CVS?

If yes and yes, then I ought to be able to have some fun finding both versions on CVS.

Here again, I think, might be the CVS version data from the script on my NetBSD-current VPS:

linveo# pwd
/etc/rc.d
linveo# head -n 4 resize_root 
#!/bin/sh
#
# $NetBSD: resize_root,v 1.6 2023/10/04 00:04:42 gutteridge Exp $
#
linveo#

Thanks!

cmeerw · December 14

@Not_Oles said:

@cmeerw said:
Ahh no, you probably got that resize_root script from NetBSD HEAD when you updated NetBSD.

@cmeerw I guess you mean that there is a newer version of the script in your NetBSD 10 image than the version of the script which is in NetBSD HEAD?

I guess you mean that the version of the script which is in your NetBSD 10 image is the same as in the latest NetBSD 10 on CVS?

No, I am patching the resize_root script when building the NetBSD image:

sed -i -e 's,\[ "$opt" = "log" \],[ "$( dumpfs -s / | fgrep "wapbl version " | cut -f1 )" != "wapbl version 0x0" ],' \
    "${MNT}/etc/rc.d/resize_root"

The original NetBSD script checks if the /etc/fstab entry contains a log option, and then refuses to resize the filesystem. But what it really should be checking is if the filesystem actually uses a journaling log (which I am doing by checking the output of dumpfs -s). This makes a difference for the first boot from the NetBSD image, as the filesystem has been created without a journaling log, but the /etc/fstab already contains the log option - this way the filesystem can be resized on the first boot, and when it then gets mounted read/write after being resized, the journaling log is correctly created.

This is basically what I have reported as PR#58723

BTW, the filesystem inconsistencies after tunefs -l 0 and an attempted re-mount is now reported as PR#58908

Not_Oles · December 14

So, if I now understand correctly, I do need to go ahead as you described previously and as quoted here:

@cmeerw said: That means you'll have to do the resize command in the shell as well (instead of relying on the resize_root script).

So I think it would then be:

reboot into single user mode
tunefs -l 0 /dev/dk2
reboot, again into single user mode
mount -u /dev/dk2 /
mount -ru /dev/dk2 /
resize_ffs /dev/dk2
reboot -n

Thanks so much for your patience with me on this!

Not_Oles · December 17

@cmeerw @linveo

@cmeerw's suggested steps seem to have worked!

Now I have to backup again! :!

@linveo Thanks again for the nice NetBSD VPS! Thanks to @cmeerw for the NetBSD 10 image!

linveo# date
Tue Dec 17 00:40:07 UTC 2024
linveo# df -h .
Filesystem     Size   Used  Avail %Cap Mounted on
/dev/dk2        96G    42G    49G  46% /
linveo#

Not_Oles · December 17

@cmeerw Not that my involvement matters, but I am watching your three NetBSD problem reports which were discussed above. I try to pay attention to all that you are teaching me. Thank you very much!

https://lowendspirit.com/discussion/comment/186451/#Comment_186451
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58693
589603: x86-64 VM: assertion "lp_max >= core_max" failed

https://lowendspirit.com/discussion/comment/198367/#Comment_198367
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58723
58723: resize_ffs should refuse to resize a filesystem with a journal

https://lowendspirit.com/discussion/comment/198367/#Comment_198367
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58908
58908: journaling log not freed when re-mounting read-only filesystem after tunefs -l 0

Not_Oles · December 17

The Wireshark build seems to have finished while I was sleeping!

-rw-r--r--   1 root  wheel   8176225 Dec 17 06:47 nohup.out

Here is the tail from make install.

  [ . . . ]
=> Checking file-check results for wireshark-4.4.1nb2
=> Creating binary package /usr/pkgsrc/net/wireshark/work/.packages/wireshark-4.4.1nb2.tgz
===> Building binary package for wireshark-4.4.1nb2
=> Creating binary package /usr/pkgsrc/packages/All/wireshark-4.4.1nb2.tgz
===> Installing binary package of wireshark-4.4.1nb2
linveo#

Wireshark indeed does seem to be installed.

linveo# ls -l $(which wireshark)
-rwxr-xr-x  1 root  wheel  10360032 Dec 17 18:08 /usr/pkg/bin/wireshark
linveo# date
Tue Dec 17 18:12:00 UTC 2024
linveo#

Here's some disk usage information, which is interesting because it suggests that the previous 50 GB disk was only a little short of space compared to the amount needed to compile Wireshark.

linveo# cd /usr
linveo# df -h .
Filesystem     Size   Used  Avail %Cap Mounted on
/dev/dk2        96G    50G    41G  54% /
linveo# du -sh pkg
3.5G    pkg
linveo# du -sh pkgsrc
33G     pkgsrc
linveo# du -sh src
3.9G    src
linveo# du -sh xsrc
1.0G    xsrc
linveo#

It seems like tshark was built and installed as well.

linveo# ls -l $(which tshark)
-rwxr-xr-x  1 root  wheel  337424 Dec 17 18:08 /usr/pkg/bin/tshark
linveo#

Looks like it's time for another backup and also to try Wireshark and tshark. I don't have any significant experience with either of them, so any hints would be appreciated. I want to look around at the sources and at the build environment and build tools.

It's beyond hilarious that @yoursunny might have been trolling when he suggested that Wireshark was my friend. Who but God can say what peoples' motivations are? But now, Wireshark seems to be here. Tshark too!

Thanks to @Linveo for the nice VPS with self-compiled NetBSD-current! Thanks to all of you guys who have been helping! Thanks to LES!

cmeerw · December 17

NetBSD 10.1 seems to be just around the corner

Not_Oles · December 17

@Not_Oles said: The Wireshark build seems to have finished while I was sleeping!

Wireshark seems to load on my Linveo VPS! Note the locale message in the xterm. That same message was shown repeatedly in the compile output. The C.UTF-8 locale seems like a great idea. In the past I seem to recall a few issues when trying the C.UTF-8 locale on other OSes. But maybe I could try C.UTF-8 on NetBSD-current.

The Wireshark menus seem to work!

It's wonderful that the entire NetBSD-current operating system and the entire userland all seems to work with everything self-compiled on a VPS! Thanks NetBSD! Thanks @Linveo! Thanks everyone!

I still have to try actually using Wireshark and tshark to do something, once I figure out how. . . .

Crab · December 18

@cmeerw said:
NetBSD 10.1 seems to be just around the corner

Do you know whether the kernel patch you provided is included?

cmeerw · December 18

@Crab said:

@cmeerw said:
NetBSD 10.1 seems to be just around the corner

Do you know whether the kernel patch you provided is included?

It's not even in HEAD. Unfortunately, there doesn't appear to be much interest in this one (even tried pointing out on the mailing list that another VPS provider is also affected).

cmeerw · December 18

@cmeerw said:
I have updated my NetBSD images to now use such an additional host route for the gateway instead of widening the subnet mask:

NetBSD 10.0 and NetBSD 9.4

@linveo could you update the templates when you get some time (and I believe to NetBSD 9.4 template was still missing the configuration meta-data from VirtFusion - should just be the same as for the 10.0 image)

I have now already made a NetBSD 10.1 image (@linveo ) as 10.1 binaries started appearing this morning.

LES BSD Thread!

Comments