Free LES Community Server from Hosteroid via MetalVPS!

Not_Oles · January 7

I heard the the filesystem code lives in the fs directory. So I took a look.

Hey! We can see ext4 in there! And xfs! And btrfs! If I understand right, the ZFS code is not present, but needs to be added via a kernel module.

root@manassas:/usr/local/src# cd linux
root@manassas:/usr/local/src/linux# ls
arch     CREDITS        fs        ipc      lib          mm      samples   tools
block    crypto         include   Kbuild   LICENSES     net     scripts   usr
certs    Documentation  init      Kconfig  MAINTAINERS  README  security  virt
COPYING  drivers        io_uring  kernel   Makefile     rust    sound
root@manassas:/usr/local/src/linux# cd fs; ls
9p                   dcache.c       fs_types.c          namei.c           remap_range.c
adfs                 debugfs        fs-writeback.c      namespace.c       romfs
affs                 devpts         fuse                netfs             select.c
afs                  direct-io.c    gfs2                nfs               seq_file.c
aio.c                dlm            hfs                 nfs_common        signalfd.c
anon_inodes.c        d_path.c       hfsplus             nfsd              smb
attr.c               drop_caches.c  hostfs              nilfs2            splice.c
autofs               ecryptfs       hpfs                nls               squashfs
backing-file.c       efivarfs       hugetlbfs           notify            stack.c
bad_inode.c          efs            init.c              nsfs.c            stat.c
bcachefs             erofs          inode.c             ntfs3             statfs.c
befs                 eventfd.c      internal.h          ocfs2             super.c
bfs                  eventpoll.c    ioctl.c             omfs              sync.c
binfmt_elf.c         exec.c         iomap               open.c            sysctls.c
binfmt_elf_fdpic.c   exfat          isofs               openpromfs        sysfs
binfmt_flat.c        exportfs       jbd2                orangefs          sysv
binfmt_misc.c        ext2           jffs2               overlayfs         tests
binfmt_script.c      ext4           jfs                 pidfs.c           timerfd.c
bpf_fs_kfuncs.c      f2fs           Kconfig             pipe.c            tracefs
btrfs                fat            Kconfig.binfmt      pnode.c           ubifs
buffer.c             fcntl.c        kernel_read_file.c  pnode.h           udf
cachefiles           fhandle.c      kernfs              posix_acl.c       ufs
ceph                 file.c         libfs.c             proc              unicode
char_dev.c           filesystems.c  lockd               proc_namespace.c  userfaultfd.c
coda                 file_table.c   locks.c             pstore            utimes.c
compat_binfmt_elf.c  freevxfs       Makefile            qnx4              vboxsf
configfs             fs_context.c   mbcache.c           qnx6              verity
coredump.c           fsopen.c       minix               quota             xattr.c
cramfs               fs_parser.c    mnt_idmapping.c     ramfs             xfs
crypto               fs_pin.c       mount.h             readdir.c         zonefs
dax.c                fs_struct.c    mpage.c             read_write.c
root@manassas:/usr/local/src/linux/fs#

Not_Oles · January 8

I hear from Google Gemini that the contents of the fs directory includes:

The Virtual File System Layer (VFS) -- abstracts common operations across various file systems to enable presenting a unified interface to user applications,
Specific File System Implementations -- btrfs, ext4, xfs, etc. -- each file systems implementation contains a "file_operations" structure with pointers to the filesystem-specific functions which implement user-level system call operations, such as read, write, open, and ioctl, and
Additional, related "helper" implementations -- e.g., read-write.c which handles certain generalized parts of read/write operations such as buffering and page-caching. For example, the functions in read-write.c might be used by multiple specific file system implementations.

I asked Google Gemini to organize the files within fs by listing the top level functions together with the files which implement each of the top level functions. Here below is Gemini's list. I commented a few places where Gemini's list didn't seem in accordance with Linus' HEAD.

Key Subsystems and Representative Files (fs/ directory):

Virtual Filesystem (VFS) Core: (Provides the common interface and infrastructure for all filesystems)
- fs/dcache.c: Directory entry cache (dentry).
- fs/inode.c: Inode management (core inode operations).
- fs/namei.c: Pathname lookup (resolving paths to inodes).
- fs/file.c: File object management (representing open files).
- fs/stat.c: File status information (stat() system call).
- fs/open.c: File opening (open() system call).
- fs/read_write.c: Core read/write operations and helpers (often used by specific filesystems). Not the primary entry point, but provides important utilities.
- fs/super.c: Superblock management (representing mounted filesystems).
- fs/vfs_kern.c: Miscellaneous VFS kernel functions. # No vfs_kern.c
- fs/libfs.c: Helper functions for various filesystem operations.
- fs/mount.c: Mounting and unmounting filesystems. # No mount.c. There is mount.h.
Specific Filesystem Implementations: (Provide the concrete implementation for each filesystem type)
- fs/ext4/: (Entire directory) Ext4 filesystem.
- fs/xfs/: (Entire directory) XFS filesystem.
- fs/btrfs/: (Entire directory) Btrfs filesystem.
- fs/fat/: (Entire directory) FAT filesystem.
- fs/ntfs/: (Entire directory) NTFS filesystem support.
- ... other filesystem directories ... (Each filesystem has its own subdirectory)
Buffer and Page Caching: (Manage caching of disk blocks and file data in memory)
- fs/buffer.c: Buffer cache management (for disk blocks).
- fs/pagemap.c: Page cache management (for file data). # No pagemap.c
- fs/bio.c: Block I/O layer. Handles submission and completion of block I/O requests. #No bio.c
Quota Management: (Limit disk usage by users and groups)
- fs/quota/: (Entire directory) Disk quota management.
File Locking: (Mechanisms for coordinating access to files)
- fs/locks.c: File locking (e.g., flock, fcntl locks).
Virtual Filesystems: (Provide filesystem-like interfaces to kernel resources)
- fs/proc/: (Entire directory) procfs (process information).
- fs/sysfs/: (Entire directory) sysfs (kernel parameters and system information).
- fs/debugfs.c: debugfs (kernel debugging).
- fs/tmpfs/: (Entire directory) tmpfs (RAM-based filesystem). # No tmpfs subdirectory
Namespaces: (Isolate filesystems and other resources)
- fs/nsfs.c: Namespace support (mount namespaces).
Other Utilities and Helpers: (Various supporting functions and modules)
- fs/ioctl.c: ioctl system calls (device-specific operations).
- fs/exec.c: Program execution (related to the execve() system call).
- fs/fs-writeback.c: Writeback mechanism (flushing dirty data to disk).
- fs/char_dev.c: Character device management (though some character device drivers might be elsewhere).
- fs/fs_struct.c: Per-process filesystem information.

I'm curious why the fs directory doesn't have subdirectories organized according to function. For example, there could be fs/vfs-core, fs/implementations, fs/caching, etc.

Apparently BSD kernels also have a Virtual Filesystem Layer ("VFS"). How early in Unix did the VFS layer appear? Was a VFS common in pre-Unix operating systems?

It's easy to imagine that ZFS conceivably might have its own implementation of some of the helper systems or even its own version of the VFS, and that these within-the-specific-filesystem implementations might have contributed to possible "difficulty" of including ZFS in the Linus kernel proper. I'm curious why ZFS wasn't included. Do we know?

Is anybody aware of a well recognized, introductory Linux kernel or BSD source code discussion which follows the "overview / top-down" style of analysis which I am trying to use here?

Is anybody up on the mistakes Google Gemini apparently made? Could these be as simple as, "Oh, that's version x.xx?"

Thanks for any help on any of these questions! Thanks for additional observations and comments!

Does anybody else want to join the server? If yes, please check How to Apply in the OP. Thanks @Hosteroid!

quangthang · January 8

Hello!
I'd like a slice for experimenting with NixOS and hosting some open-source projects (Vaultwarden, Whoogle search, Openwebui). If you need any additional infos about me, I'll happy to provide through PM.
My ssh public key is ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHZ7KokkDS4XU9M15R3htHbt4ZJ9NQeYxVbKWinbE3n5.
Thank you!

hobofl · January 8

@Not_Oles said: I'm curious why ZFS wasn't included. Do we know?

Not being that familiar with the history of the relevant discussions I don't know if there are also other (technical, political, social, whatever) reasons, but the reason that is usually given is the legal uncertainty of GPLv2 (the Linux license) and CDDL (the ZFS license) compatibility. Here's one summary, with links to other discussion (including opposing opinions):

https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/

Not_Oles · January 8

@quangthang said:
Hello!
I'd like a slice for experimenting with NixOS and hosting some open-source projects (Vaultwarden, Whoogle search, Openwebui). If you need any additional infos about me, I'll happy to provide through PM.
My ssh public key is ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHZ7KokkDS4XU9M15R3htHbt4ZJ9NQeYxVbKWinbE3n5.
Thank you!

Hi @quangthang!

Thanks for your request, which makes me happy! Of course, I definitely do remember you from previous servers, and so I am very delighted to have you joining hlcs!

I sent you login info. You are sharing root with me and with @cmeerw.

Please feel free to do whatever you want as long as it's White Hat. Please post here about what you are doing and about any questions you have.

Always best wishes!

Tom

Thanks to @Hosteroid for our very nice server!

quangthang · 9:00AM

Hello everyone.
I just got around to do some basic setup (set up user account, change to fish shell).

BashVM is installed

It seems like it's not installed globally. I cloned the repo into my home dir, ran the install script again but it mostly skipping cause required pkgs and configs are already installed

...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

dnsmasq is already installed.

Default storage pool is already active.

Default network is already active.
...

I'm gonna go ahead and see if I can make a NixOS VM.

quangthang · 9:28AM

VM created, now to the installation. Not sure if it's my part but i'm getting constant vnc timed out

quangthang · 9:51AM

The installation process was successful but it's stuck on this screen. I've must have done something wrong. I think I'll grab the graphical iso to use their graphical installer.

quangthang · 10:10AM

It's up!

quangthang · 3:07PM

One of dependencies on my config keep failing build. Which is weird cause it built just fine in other instances and my local laptop. Anyway, I'm going to call it a day.

cmeerw · 3:30PM

@quangthang said:

One of dependencies on my config keep failing build. Which is weird cause it built just fine in other instances and my local laptop. Anyway, I'm going to call it a day.

Isn't that just a timing thing, i.e. it took 6 ms too long?

Not_Oles · 9:55PM

@hobofl said:

@Not_Oles said: I'm curious why ZFS wasn't included. Do we know?

Not being that familiar with the history of the relevant discussions I don't know if there are also other (technical, political, social, whatever) reasons, but the reason that is usually given is the legal uncertainty of GPLv2 (the Linux license) and CDDL (the ZFS license) compatibility. Here's one summary, with links to other discussion (including opposing opinions):

https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/

Hi @hobofl!

Sorry about my delayed response to your comment. I wanted to wait until I had a chance to look at the SFConservancy link you provided.

I really appreciate your comment for at least four reasons.

First, your comment concisely and directly answers the question I asked and not some other question.
Second, your comment provides excellent context: you sketched a typology of possible reasons why some software project might or might not be included in another software project: licensing, technical, social, political, other.
Third, your comment includes a link to relevant and comprehensive further discussion from multiple, inconsistent points-of-view.
Fourth, the tone of your comment is friendly and positive and helpful instead of negative and snarky.

Now that, thanks to you, I might understand a little better, I have to decide what to do.

It seems like Debian and the Conservancy and Oracle might have arrived, for the time being, at somewhat of a compromise regarding the use of ZFS on Linux. I guess that compromise might be good enough for me to continue glancing at ZFS on Debian on our Community Server. Nevertheless, I now imagine that I might want a really important reason before continuing with ZFS beyond an educational glance. I might want some specific context where ZFS clearly would be a better choice than any of the other available file systems.

Thanks again for your really helpful comment! Thanks to @Hosteroid for our beautiful Community Server!

Not_Oles · 10:05PM

Hi @quangthang! I've seen your posts above but I haven't had a chance to read them carefully. Luckily @cmeerw was here and kindly responded! I'm glad you got in to the server! Yaaay!

Free LES Community Server from Hosteroid via MetalVPS!

Comments