Duh! If I restore from the NetBSD backup, is that going to take the file space back to 50G? In which case I would have to start again at the beginning with @cmeerw's image? We will see. Maybe.
@Not_Oles said:
Duh! If I restore from the NetBSD backup, is that going to take the file space back to 50G? In which case I would have to start again at the beginning with @cmeerw's image? We will see. Maybe.
For now, I am back in to the NetBSD restored from the backup.
linveo# df
Filesystem 1K-blocks Used Avail %Cap Mounted on
/dev/dk2 50313000 44380778 3416572 92% /
kernfs 1 1 0 100% /kern
ptyfs 1 1 0 100% /dev/pts
procfs 4 4 0 100% /proc
tmpfs 1048440 4 1048436 0% /tmp
tmpfs 1048440 4 1048436 0% /var/shm
linveo# cat /etc/fstab
# NetBSD /etc/fstab
# See /usr/share/examples/fstab/ for more examples.
NAME=netbsd-root / ffs rw,log 1 1
NAME=netbsd-swap none swap sw 0 0
kernfs /kern kernfs rw
ptyfs /dev/pts ptyfs rw
procfs /proc procfs rw
/dev/cd0a /cdrom cd9660 ro,noauto
tmpfs /tmp tmpfs rw,-m1777,-sram%25
tmpfs /var/shm tmpfs rw,-m1777,-sram%25
linveo# man resize_ffs
[ . . . ]
EXAMPLES
resize_ffs /dev/vg00/rlv1
will enlarge the file system on the Logical Volume /dev/vg00/lv1 from
Volume Group vg00 to the current device size.
[ . . . ]
linveo# resize_ffs /dev/dk2
It's required to manually run fsck on file system before you can resize it
Did you run fsck on your disk (Yes/No) ? No
Nothing done
linveo# man fsck
[ . . . ]
linveo# fsck /dev/dk2
fsck: cannot open `/dev/dk2': Device busy
linveo#
From reading the resize_ffs man page, it seems like it's not really "required" to run fsck. Instead maybe I could just use resize_ffs' -y option?
But, to run fsck, I might have to unmount /dev/dk2? If I unmount /dev/dk2, how do I retain access to the fsck binary which lives on /dev/dk2 in /sbin? Is fsck static? Do I copy the fsck binary to /tmp?
Hmm. There's an fsck which lives in /rescue. I never used the NetBSD rescue system before.
I have to look around more and read more. Additional hints are welcome! Thanks @Crab!
So the main issue is that resize_ffs doesn't really handle the filesystem log, so resizing such a filesystem is disabled in my boot scripts. In some cases you could run into data corruption (although that would be with an out-of-filesystem log)
The clean way to do it is to clear the log (see tunefs -l 0 - which actually means to set a flag to clear the log, and mount the filesystem that then clears the log), resize the filesystem, and the log will then be recreated on the next mount. It's just painful to do that on the root filesystem (as that can't be completely unmounted).
Alternatively, you could just disable the check in /etc/rc.d/resize_root and let it resize the filesystem anyway (I think, it will just mean that the log will not be resized)
@Not_Oles said: Thanks! I will try to try it tomorrow. Sleep now. Zzzzz!
This should work:
reboot into single user mode
tunefs -l 0 /dev/dk2
mount -ruo reload /dev/dk2 /
reboot (this should then resize the filesystem, and trigger another reboot)
in between these commands you can check with dumpfs -s / the wapbl lines (tunefs should set the flags to 0x2), and after the mount with reload, wapbl version should be 0x0 and the wapbl loc entries should all be 0 again.
@Not_Oles said: Thanks! I will try to try it tomorrow. Sleep now. Zzzzz!
This should work:
reboot into single user mode
tunefs -l 0 /dev/dk2
mount -ruo reload /dev/dk2 /
reboot (this should then resize the filesystem, and trigger another reboot)
in between these commands you can check with dumpfs -s / the wapbl lines (tunefs should set the flags to 0x2), and after the mount with reload, wapbl version should be 0x0 and the wapbl loc entries should all be 0 again.
And now that I have figured out the exact commands of how to do it, I could probably even integrate that into the resize_root script.
I tried them. Everything seemed to go as expected. However, a message flashed by during the first reboot. It said something like, "Not resizing logging not enabled." Then the second reboot went ahead, and succeeded, but the disk size seemed not to have increased. Do we need to enable logging?
Remember that I might have made a mistake somewhere. It's not necessarily that your steps are wrong. For example, I chose /bin/sh for the shell. Should I have chosen /rescue/sh?
@Not_Oles those kernel messages don't look like they should be expected (maybe that's a change with NetBSD HEAD?)
BTW, Not resizing $rootmp: logging unsupported should be the message you are seeing (and the tunefs, re-mount was supposed to disable logging while resizing)
@cmeerw said: @Not_Oles those kernel messages don't look like they should be expected (maybe that's a change with NetBSD HEAD?)
BTW, Not resizing $rootmp: logging unsupported should be the message you are seeing (and the tunefs, re-mount was supposed to disable logging while resizing)
Ok... while it seemed to work with NetBSD 10.0 for me, it actually left the old journal unconnected on the filesystem, so that's not too good. You might want to do a fsck_ffs -f /dev/dk2 in single user mode now.
@cmeerw said: @Not_Oles those kernel messages don't look like they should be expected (maybe that's a change with NetBSD HEAD?)
BTW, Not resizing $rootmp: logging unsupported should be the message you are seeing (and the tunefs, re-mount was supposed to disable logging while resizing)
Ok... while it seemed to work with NetBSD 10.0 for me, it actually left the old journal unconnected on the filesystem, so that's not too good. You might want to do a fsck_ffs -f /dev/dk2 in single user mode now.
@cmeerw said: Not resizing $rootmp: logging unsupported should be the message you are seeing
That looks like what I saw. But I am not sure because it scrolled by so quickly. Are messages like this one recorded somewhere?
Some of the output from startup scripts might end up in /var/log/messages, but I don't think this one does (that particular script uses # KEYWORD: interactive, so the output doesn't get redirected to the syslog processor).
@cmeerw said: left the old journal unconnected on the filesystem
May I please ask how you figured this out? Thanks!
The kernel messages from HEAD don't look very nice, so I thought I'd just do a fsck in case, and it added those journals to lost+found (so the mount with reload must have managed to just set the information in the superblock to 0, but then didn't clear the inodes as the filesystem was supposed to be only mounted read-only).
linveo# pwd
/etc/rc.d
linveo# cat -n resize_root
[ . . . ]
68 for opt in $(split_options "${fs_mntops}"); do
69 if [ "$opt" = "log" ]; then
70 echo "Not resizing $rootmp: logging unsupported"
71 return
72 fi [ . . . ]
I don't think I understood much of the sh code, but it was fun to be able to find quickly and easily what might seem to be the code where the message we are discussing originated.
linveo# pwd
/usr/src/sbin/resize_ffs
linveo# ls
CVS Makefile TODO resize_ffs.8 resize_ffs.c
linveo#
I don't think I understood much of the C code, but I really appreciated the helpful comments in resize_ffs.c. It seems like mouse really wants people to be able to understand his code!
Thanks to @cmeerw, @linveo, and the NetBSD developers! And LES!
linveo# pwd
/etc/rc.d
linveo# cat -n resize_root
[ . . . ]
68 for opt in $(split_options "${fs_mntops}"); do
69 if [ "$opt" = "log" ]; then
70 echo "Not resizing $rootmp: logging unsupported"
71 return
72 fi [ . . . ]
Oh... that's not good. That version of the script is a bit older than I thought (I though you already had the version that did the check via dumpfs instead of the flag in /etc/fstab)
That means you'll have to do the resize command in the shell as well (instead of relying on the resize_root script).
linveo# pwd
/etc/rc.d
linveo# cat -n resize_root
[ . . . ]
68 for opt in $(split_options "${fs_mntops}"); do
69 if [ "$opt" = "log" ]; then
70 echo "Not resizing $rootmp: logging unsupported"
71 return
72 fi [ . . . ]
Oh... that's not good. That version of the script is a bit older than I thought (I though you already had the version that did the check via dumpfs instead of the flag in /etc/fstab)
That means you'll have to do the resize command in the shell as well (instead of relying on the resize_root script).
So I think it would then be:
reboot into single user mode
tunefs -l 0 /dev/dk2
reboot, again into single user mode
mount -u /dev/dk2 /
mount -ru /dev/dk2 /
resize_ffs /dev/dk2
reboot -n
Ahh no, you probably got that resize_root script from NetBSD HEAD when you updated NetBSD.
@cmeerw said:
Ahh no, you probably got that resize_root script from NetBSD HEAD when you updated NetBSD.
@cmeerw I guess you mean that there is a newer version of the script in your NetBSD 10 image than the version of the script which is in NetBSD HEAD?
I guess you mean that the version of the script which is in your NetBSD 10 image is the same as in the latest NetBSD 10 on CVS?
No, I am patching the resize_root script when building the NetBSD image:
sed -i -e 's,\[ "$opt" = "log" \],[ "$( dumpfs -s / | fgrep "wapbl version " | cut -f1 )" != "wapbl version 0x0" ],' \
"${MNT}/etc/rc.d/resize_root"
The original NetBSD script checks if the /etc/fstab entry contains a log option, and then refuses to resize the filesystem. But what it really should be checking is if the filesystem actually uses a journaling log (which I am doing by checking the output of dumpfs -s). This makes a difference for the first boot from the NetBSD image, as the filesystem has been created without a journaling log, but the /etc/fstab already contains the log option - this way the filesystem can be resized on the first boot, and when it then gets mounted read/write after being resized, the journaling log is correctly created.
This is basically what I have reported as PR#58723
BTW, the filesystem inconsistencies after tunefs -l 0 and an attempted re-mount is now reported as PR#58908
So, if I now understand correctly, I do need to go ahead as you described previously and as quoted here:
@cmeerw said: That means you'll have to do the resize command in the shell as well (instead of relying on the resize_root script).
So I think it would then be:
reboot into single user mode
tunefs -l 0 /dev/dk2
reboot, again into single user mode
mount -u /dev/dk2 /
mount -ru /dev/dk2 /
resize_ffs /dev/dk2
reboot -n
@cmeerw Not that my involvement matters, but I am watching your three NetBSD problem reports which were discussed above. I try to pay attention to all that you are teaching me. Thank you very much!
https://lowendspirit.com/discussion/comment/186451/#Comment_186451
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58693
589603: x86-64 VM: assertion "lp_max >= core_max" failed
https://lowendspirit.com/discussion/comment/198367/#Comment_198367
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58723
58723: resize_ffs should refuse to resize a filesystem with a journal
https://lowendspirit.com/discussion/comment/198367/#Comment_198367
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58908
58908: journaling log not freed when re-mounting read-only filesystem after tunefs -l 0
Comments
Moving to Debian got the bump to 100G.
Duh! If I restore from the NetBSD backup, is that going to take the file space back to 50G? In which case I would have to start again at the beginning with @cmeerw's image? We will see. Maybe.
I hope everyone gets the servers they want!
man resize_ffs
Yes.
Thanks @Crab!
Will check the resize_ffs man page.
For now, I am back in to the NetBSD restored from the backup.
From reading the resize_ffs man page, it seems like it's not really "required" to run fsck. Instead maybe I could just use resize_ffs' -y option?
But, to run fsck, I might have to unmount /dev/dk2? If I unmount /dev/dk2, how do I retain access to the fsck binary which lives on /dev/dk2 in /sbin? Is fsck static? Do I copy the fsck binary to /tmp?
Hmm. There's an fsck which lives in /rescue. I never used the NetBSD rescue system before.
I have to look around more and read more. Additional hints are welcome! Thanks @Crab!
I hope everyone gets the servers they want!
Can you enable VNC, reboot and add '-a' boot flag and put '/rescue/init' as the init path. All the tools at /rescue are statically linked.
Also https://www.netbsd.org/docs/misc/index.html#using-fsck
Thanks! I will try to try it tomorrow. Sleep now. Zzzzz!
@Crab You're a nice guy!
Thanks again! 
I hope everyone gets the servers they want!
So the main issue is that resize_ffs doesn't really handle the filesystem log, so resizing such a filesystem is disabled in my boot scripts. In some cases you could run into data corruption (although that would be with an out-of-filesystem log)
The clean way to do it is to clear the log (see
tunefs -l 0- which actually means to set a flag to clear the log, and mount the filesystem that then clears the log), resize the filesystem, and the log will then be recreated on the next mount. It's just painful to do that on the root filesystem (as that can't be completely unmounted).Alternatively, you could just disable the check in
/etc/rc.d/resize_rootand let it resize the filesystem anyway (I think, it will just mean that the log will not be resized)This should work:
tunefs -l 0 /dev/dk2mount -ruo reload /dev/dk2 /reboot(this should then resize the filesystem, and trigger another reboot)in between these commands you can check with
dumpfs -s /thewapbllines (tunefsshould set the flags to0x2), and after the mount withreload,wapbl versionshould be0x0and thewapbl locentries should all be0again.And now that I have figured out the exact commands of how to do it, I could probably even integrate that into the
resize_rootscript.Hi @cmeerw!
Thanks for the suggested steps!
I tried them. Everything seemed to go as expected. However, a message flashed by during the first reboot. It said something like, "Not resizing logging not enabled." Then the second reboot went ahead, and succeeded, but the disk size seemed not to have increased. Do we need to enable logging?
Remember that I might have made a mistake somewhere. It's not necessarily that your steps are wrong. For example, I chose /bin/sh for the shell. Should I have chosen /rescue/sh?
More ideas, please? Thank you again so much!
Tom
I hope everyone gets the servers they want!
fdisk -l ?
Has the partition been extended?
Just a couple of thoughts.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
@Not_Oles those kernel messages don't look like they should be expected (maybe that's a change with NetBSD HEAD?)
BTW,
Not resizing $rootmp: logging unsupportedshould be the message you are seeing (and thetunefs, re-mountwas supposed to disable logging while resizing)Ok... while it seemed to work with NetBSD 10.0 for me, it actually left the old journal unconnected on the filesystem, so that's not too good. You might want to do a
fsck_ffs -f /dev/dk2in single user mode now.Alternative approach:
tunefs -l 0 /dev/dk2mount -u /dev/dk2 /mount -ru /dev/dk2 /exitThat looks like what I saw. But I am not sure because it scrolled by so quickly. Are messages like this one recorded somewhere?
May I please ask how you figured this out? Thanks!
Will do!
Will try.
Thanks so much for helping so much!
Thanks again to Linveo for the nice VPS!
I hope everyone gets the servers they want!
Some of the output from startup scripts might end up in
/var/log/messages, but I don't think this one does (that particular script uses# KEYWORD: interactive, so the output doesn't get redirected to the syslog processor).The kernel messages from HEAD don't look very nice, so I thought I'd just do a fsck in case, and it added those journals to
lost+found(so themountwithreloadmust have managed to just set the information in the superblock to0, but then didn't clear the inodes as the filesystem was supposed to be only mounted read-only).I had some fun looking at
I don't think I understood much of the sh code, but it was fun to be able to find quickly and easily what might seem to be the code where the message we are discussing originated.
I don't think I understood much of the C code, but I really appreciated the helpful comments in resize_ffs.c. It seems like mouse really wants people to be able to understand his code!
Thanks to @cmeerw, @linveo, and the NetBSD developers!
And LES! 
I hope everyone gets the servers they want!
Okay, I see this in line 8. Thanks again!
I hope everyone gets the servers they want!
Oh... that's not good. That version of the script is a bit older than I thought (I though you already had the version that did the check via
dumpfsinstead of the flag in/etc/fstab)That means you'll have to do the resize command in the shell as well (instead of relying on the
resize_rootscript).So I think it would then be:
tunefs -l 0 /dev/dk2mount -u /dev/dk2 /mount -ru /dev/dk2 /resize_ffs /dev/dk2reboot -nAhh no, you probably got that
resize_rootscript from NetBSD HEAD when you updated NetBSD.@cmeerw I guess you mean that there is a newer version of the script in your NetBSD 10 image than the version of the script which is in NetBSD HEAD?
I guess you mean that the version of the script which is in your NetBSD 10 image is the same as in the latest NetBSD 10 on CVS?
If yes and yes, then I ought to be able to have some fun finding both versions on CVS.
Here again, I think, might be the CVS version data from the script on my NetBSD-current VPS:
Thanks!
I hope everyone gets the servers they want!
No, I am patching the resize_root script when building the NetBSD image:
The original NetBSD script checks if the
/etc/fstabentry contains alogoption, and then refuses to resize the filesystem. But what it really should be checking is if the filesystem actually uses a journaling log (which I am doing by checking the output ofdumpfs -s). This makes a difference for the first boot from the NetBSD image, as the filesystem has been created without a journaling log, but the/etc/fstabalready contains thelogoption - this way the filesystem can be resized on the first boot, and when it then gets mounted read/write after being resized, the journaling log is correctly created.This is basically what I have reported as PR#58723
BTW, the filesystem inconsistencies after
tunefs -l 0and an attempted re-mount is now reported as PR#58908So, if I now understand correctly, I do need to go ahead as you described previously and as quoted here:
Thanks so much for your patience with me on this!
I hope everyone gets the servers they want!
@cmeerw @linveo
@cmeerw's suggested steps seem to have worked!
Now I have to backup again! :!
@linveo Thanks again for the nice NetBSD VPS!
Thanks to @cmeerw for the NetBSD 10 image! 
I hope everyone gets the servers they want!
@cmeerw Not that my involvement matters, but I am watching your three NetBSD problem reports which were discussed above. I try to pay attention to all that you are teaching me. Thank you very much!
I hope everyone gets the servers they want!