Intel iGPU VAAPI in Unprivileged LXC 4.0 Container

yoursunny · February 2022

This article is originally published on yoursunny.com blog https://yoursunny.com/t/2022/lxc-vaapi/

Background

I recently bought a DELL OptiPlex 7040 Micro desktop computer and wanted to operate it as a dedicated server.
I installed Debian 11 on the computer, and placed it into the closet to be accessed over SSH only.
To keep the host machine stable, I decide to run most workloads in LXC containers, which are said to be Fast-as-Metal.
Since I operate my own video streaming website, I have an LXC container for encoding the videos.

The computer comes with an Intel Core i5-6500T processor.
It has 4 hardware cores running at 2.50GHz frequency, and belongs to the Skylake family.
FFmpeg is happily encoding my videos on this CPU.

As I read through the processor specification, I noticed this section:

Processor Graphics: Intel® HD Graphics 530
- Processor Graphics indicates graphics processing circuitry integrated into the processor, providing the graphics, compute, media, and display capabilities.
Intel® Quick Sync Video: Yes
- Intel® Quick Sync Video delivers fast conversion of video for portable media players, online sharing, and video editing and authoring.

It seems that I have a GPU!
Can I make use of this Intel GPU and accelerate video encoding workloads?

Story

If you just want the solution, skip to the TL;DR Steps to Enable VAAPI in LXC section at the end.

Testing VAAPI with Docker

I read FFmpeg HWAccelIntro and QuickSync pages, and learned:

FFmpeg supports hardware acceleration on various GPU brands including Intel, AMD, and NVIDIA.
Hardware encoders typically generate outputs of significantly lower quality than good software encoders, but are generally faster and do not use much CPU resource.
On Linux, FFmpeg may access Intel GPU through libmfx, OpenCL, or VAAPI.
Among these, encoding is possible with libmfx or VAAPI.
Each generation Intel processors has different video encoding capabilities.
For the Skylake family that I have, the integrated GPU can encode to H.264, MPEG-2, VP8, and H.265 formats.

I decided to experiment with VAAPI, because it has the shortest name 🤪.
I quickly found jrottenberg/ffmpeg Docker image.
Following the example commands on FFmpeg VAAPI page, I verified that my GPU can successfully encode videos to H264 format:

docker run \
    --device /dev/dri \
    -v $(pwd):/data -w /data \
  jrottenberg/ffmpeg:4.1-vaapi \
    -loglevel info -stats \
    -vaapi_device /dev/dri/renderD128 \
    -i input.mov \
    -vf 'hwupload,scale_vaapi=w=640:h=480:format=nv12' \
    -preset ultrafast \
    -c:v h264_vaapi \
    -f mp4 output.mp4

The renderD128 Device

This above docker run command tells me that the /dev/dri/renderD128 device is likely the key of getting Intel GPU to work in an LXC container.
It is a character device with major number 226 and minor number 128.

sunny@sunnyD:~$ ls -l /dev/dri
total 0
drwxr-xr-x 2 root root         80 Jan 22 11:04 by-path
crw-rw---- 1 root video  226,   0 Jan 22 11:04 card0
crw-rw---- 1 root render 226, 128 Jan 22 11:04 renderD128

Inside the container, this device does not exist.
Naively, I tried mknod, but it returns an "operation not permitted" error:

ubuntu@video:~$ ls -l /dev/dri
ls: cannot access '/dev/dri': No such file or directory

ubuntu@video:~$ sudo mkdir /dev/dri

ubuntu@video:~$ sudo mknod /dev/dri/renderD128 c 226 128
mknod: /dev/dri/renderD128: Operation not permitted

I searched for this problem over several weeks, found several articles regarding how to get Plex or Emby media server to use VAAPI hardware encoding from LXC containers, but they are either using Proxmox or LXD (unavailable on Debian), both differ from the plain LXC that I'm trying to use.
From these articles, I gathered enough hints on what's needed:

LXC container cannot mknod arbitrary devices for security reasons.
To have a device inode in an LXC container, the container config must:
- grant permission with lxc.cgroup.devices.allow directive, and
- mount the device with lxc.mount.entry directory.
In addition to ffmpeg, it's necessary to install vainfo i965-va-driver packages (available on both Debian and Ubuntu).

nobody:nogroup

With these configs in place, the device showed up in the container, but it does not work:

ubuntu@video:~$ ls -l /dev/dri
total 0
crw-rw---- 1 nobody nogroup 226, 128 Jan 22 16:04 renderD128
ubuntu@video:~$ vainfo
error: can't connect to X server!
error: failed to initialize display
ubuntu@video:~$ sudo vainfo
error: XDG_RUNTIME_DIR not set in the environment.
error: can't connect to X server!
error: failed to initialize display

One suspicious thing is the nobody:nogroup owner on the renderD128 device.
It differs from the root:render owner as seen on the host machine.
Naively, I tried chown, but it returns an "invalid argument" error and has no effect:

ubuntu@video:~$ sudo chown root:render /dev/dri/renderD128
chown: changing ownership of '/dev/dri/renderD128': Invalid argument

ubuntu@video:~$ ls -l /dev/dri
total 0
crw-rw---- 1 nobody nogroup 226, 128 Jan 22 16:04 renderD128

A Reddit post claims that running chmod 0666 /dev/dri/renderD128 from the host machine would solve this problem.
I gave it a try and it was indeed effective.
However, I know this isn't a proper solution because you are not supposed to change permission on device inodes.
So I continued searching.

idmap

The last piece of the puzzle lies in user and group ID mappings.
In an unprivileged LXC container, user and group IDs are shifted, so that the root user (UID 0) inside the container would not gain root privilege on the host machine.
lxc.idmap directive in the container config controls these mappings.
In my container, the relevant config was:

# map container UID 0~65535 to host UID 100000~165535
lxc.idmap = u 0 100000 65536
# map container GID 0~65535 to host GID 100000~165535
lxc.idmap = g 0 100000 65536

Notably, the root user (UID 0) and render group (GID 107) on the host user aren't mapped to anything in the container.
The kernel uses 65534 to represent a UID/GID which is outside the container's map.
Hence, the renderD128 device, when mounted into the container, has owner UID and GID being 65534:

ubuntu@video:~$ ls -ln /dev/dri
total 0
crw-rw---- 1 65534 65534 226, 128 Jan 22 16:04 renderD128

65534 is the UID of nobody and the GID of nogroup, which is why this device appears to be owned by nobody:nogroup.

To make the renderD128 owned by render group, the correct solution is mapping the render group inside the container to the render group on the host.
This, in turn, requires two ingredients:

/etc/subgid must authorize the host user who starts the container to map the GID of the host's render group into child namespaces.
The container config should have an lxc.idmap directive that maps the GID of the container's render group to the GID of the host's render group.

So I added lxc:107:1 to /etc/subgid, in which lxc is the ordinary user on the host machine that starts the containers, and 107 is the GID of render group on the host machine.
Then I modified the container config as:

# map container UID 0-65535 to host UID 100000-165535
lxc.idmap = u 0 100000 65536
# map container GID 0-65535 to host GID 100000-165535
lxc.idmap = g 0 100000 65536
# map container GID 109 to host GID 107
lxc.idmap = g 109 107 1

However, the container fails to start:

lxc@sunnyD:~$ lxc-unpriv-start -F video
Running scope as unit: run-r611f1778b87645918a2255d44073b86b.scope
lxc-start: video: conf.c: lxc_map_ids: 2865 newgidmap failed to write mapping "newgidmap: write to gid_map failed: Invalid argument": newgidmap 5297 0 100000 65536 109 107 1
             lxc-start: video: start.c: lxc_spawn: 1726 Failed to set up id mapping.

Re-reading user_namespaces(7) manpage reveals the reason:

Defining user and group ID mappings: writing to uid_map and gid_map

The range of user IDs (group IDs) specified in each line cannot overlap with the ranges in any other lines.

The above container config defines two group ID mappings that overlaps at the GID 109, which causes the failure.
Instead, it must be split to three ranges: 0-108 mapped to 100000-100108, 109 mapped to 107, 110-65535 mapped to 100110-165535.

Another idea I had, changing the GID of the render group to a large number greater than 65535 and thus dodge the overlap, turns out to be a bad idea, as it causes an error during system upgrades:

ubuntu@video:~$ sudo apt full-upgrade
Setting up udev (245.4-4ubuntu3.15) ...
The group `render' already exists and is not a system group. Exiting.
dpkg: error processing package udev (--configure):
 installed udev package post-installation script subprocess returned error exit status 1

Hence, I must carefully calculate the GID ranges and write three GID mapping entries.
With this final piece in place, success!

ubuntu@video:~$ vainfo 2>/dev/null | head -10
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 2.4.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP

Encoding speed comparison on one of my videos:

h264, ultrafast, 640x480 resolution

Intel GPU VAAPI encoding:

frame= 2900 fps=201 q=-0.0 Lsize=   18208kB time=00:01:36.78 bitrate=1541.2kbits/s speed=6.71x
video:16583kB audio:1528kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.533910%

Skylake CPU encoding:

frame= 2900 fps=171 q=-1.0 Lsize=   18786kB time=00:01:36.78 bitrate=1590.1kbits/s speed=5.71x
video:17177kB audio:1528kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.434900%

GPU encoding was 17.5% faster than CPU encoding.

TL;DR Steps to Enable VAAPI in LXC

Confirm that the /dev/dri/renderD128 device exists on the host machine.
```
lxc@sunnyD:~$ ls -l /dev/dri/renderD128
crw-rw---- 1 root render 226, 128 Jan 22 11:04 /dev/dri/renderD128
```
If the device does not exist, you do not have an Intel GPU or it is not recognized by the kernel.
You must resolve this issue before proceeding to the next step.
Find the GID of the render group on the host machine:
```
lxc@sunnyD:~$ getent group render
render:x:107:
```
On my computer, the GID is 107.
Authorize the host user who starts LXC containers to map the GID to child namespaces.
1. Run sudoedit /etc/subgid to open the editor.
2. Append a line:
```
lxc:107:1
```
Explanation:
- lxc refers to the host user account.
- 107 is the GID of the render group, as seen in step 2.
- 1 means authorizing just one GID.

Create and start an LXC container, and find out the GID of the container's render group.
I'm using a Ubuntu 20.04 template, but the same procedure is applicable to other templates.

lxc@sunnyD:~$ export DOWNLOAD_KEYSERVER=keyserver.ubuntu.com

lxc@sunnyD:~$ lxc-create -n video -t download -- -d ubuntu -r focal -a amd64
Using image from local cache
Unpacking the rootfs

You just created an Ubuntu focal amd64 (20211228_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

lxc@sunnyD:~$ lxc-unpriv-start video
Running scope as unit: run-re7a88541bd5d42ab92c9ea6d4cd2a19f.scope

lxc@sunnyD:~$ lxc-unpriv-attach video getent group render
Running scope as unit: run-reaad3e4a549a420bacb160fd8cbc87a8.scope
render:x:109:

Edit the container config.
1. Run editor ~/.local/share/lxc/video/config to open the editor.
2. Delete existing lines that start with lxc.idmap = g.
  
  However, do not delete lines that start with lxc.idmap = u.
3. Append these lines:
```
lxc.idmap = g 0 100000 109
lxc.idmap = g 109 107 1
lxc.idmap = g 110 100110 65426
lxc.cgroup.devices.allow = c 226:128 rwm
lxc.mount.entry = /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
```
Explanation:
- The lxc.idmap = g directive defines a group ID mapping.
  - 109 is the GID of the container's render group, as seen instep 4.
  - 107 is the GID of the host's render group, as seen in step 2.
- The lxc.cgroup.devices.allow directive exposes a device to the container.
  - 226:127 is the major number and minor number of the renderD128 device, as seen in step 1.
- The lxc.mount.entry directive mounts the host's renderD128 device into the container.
You may use this handy idmap calculator to generate the lxc.idmap directives:
(read original article https://yoursunny.com/t/2022/lxc-vaapi/ to use the JavaScript calculator)

Restart the container and attach to its console.

lxc@sunnyD:~$ lxc-stop video

lxc@sunnyD:~$ lxc-unpriv-start video
Running scope as unit: run-r77f46b8ba5b24254a99c1ef9cb6384c3.scope

lxc@sunnyD:~$ lxc-unpriv-attach video
Running scope as unit: run-r11cf863c81e74fcfa1615e89902b1284.scope

Install FFmpeg and VAAPI packages in the container.

root@video:/# apt update

root@video:/# apt install --no-install-recommends ffmpeg vainfo i965-va-driver
0 upgraded, 148 newly installed, 0 to remove and 15 not upgraded.
Need to get 79.2 MB of archives.
After this operation, 583 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Confirm that the /dev/dri/renderD128 device exists in the container and is owned by render group.
```
root@video:/# ls -l /dev/dri/renderD128
crw-rw---- 1 nobody render 226, 128 Jan 22 16:04 /dev/dri/renderD128
```
It's normal for the owner user to show as nobody.
This does not affect operation as long as the calling user is a member of the render group.
The only implication is that, the container's root user cannot access the renderD128 unless it is added to the render group.

Add container's user account(s) to render group.
These users will have access to the GPU.

root@video:/# /sbin/adduser ubuntu render
Adding user `ubuntu' to group `render' ...
Adding user ubuntu to group render
Done.

Become one of these users, and verify the Intel iGPU is operational in the LXC container.

root@video:/# sudo -iu ubuntu

ubuntu@video:~$ vainfo
error: XDG_RUNTIME_DIR not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: va_openDriver() returns -1
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 2.4.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointStats
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointStats
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointStats
      VAProfileH264MultiviewHigh      : VAEntrypointVLD
      VAProfileH264MultiviewHigh      : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileH264StereoHigh         : VAEntrypointEncSlice
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice

Conclusion

This article explores how to make use of Intel processor's integrated GPU in an unprivileged LXC 4.0 container, on Debian 11 bullseye host machine without Proxmox or LXD.
The key points include mounting the renderD128 device into the container, configuring idmap for the render group, and verifying the setup with vainfo command.
The result is an LXC container that can encode videos to H.264 and other formats in the GPU with Intel Quick Sync Video feature, which is 17.5% faster than CPU encoding.

MikeA · February 2022

This is great. Even though I half-assed read only half of it.