systemd generator script broken with systemd 256~rc3

[gibmat]

Over the weekend I updated one of my Debian sid containers, which pulled in updated systemd packages (256~rc3). After that update, several services are now failing with status=243/CREDENTIALS. On the host I'm seeing apparmor messages like this:

audit: type=1400 audit(1716334420.832:49): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="incus-user-1000_neat-elk_</var/lib/incus>" name="/dev/shm/" pid=1892 comm="(sd-mkdcreds)" flags="ro, nosuid, nodev, noexec, remount, bind"

I haven't had time to look into this myself, but did find two LXD bugs that look like they might be this same issue: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2046486 and canonical/lxd#12698.

I've seen this on both bookworm and sid hosts, running Debian's 6.0.0 package of Incus.

[stgraber]

So that's a bit of a mess...

Your denial shows:

flags="ro, nosuid, nodev, noexec, remount, bind"

Which happens to be a combination of flags we already allow:

https://github.com/lxc/incus/blob/main/internal/server/apparmor/instance_lxc.go#L539

I'm not seeing any mention of that nosymfollow in your case in the kernel log and because apparmor in stable (bookworm) doesn't support it, there isn't really anything we can do...

[stgraber]

Going to do a test on Ubuntu 24.04 as that's the only distro that seems to ship with AppArmor 4 and it's shipping a pre-release version for that matter...

[stgraber]

AppArmor is so broken... I've confirmed that adding nosymfollow indeed works, but the kernel code never logs it as being needed...

[gibmat]

Is there any sort of workaround we could apply for everyone who's still got some apparmor 3.x version on the host side? (Short of totally disabling a container's apparmor profile, that is.)

As more distros pickup newer versions of systemd I think this will become a wider issue, especially since apparmor 4.0 hasn't had an official release yet. Specifically for my sid container, systemd-resolved doesn't start which then borks pretty much all network connectivity for that container.

[stgraber]

I suspect security.nesting=true should work as it allows all mounts.

[gibmat]

Ah, this was a bit of red herring; I've found the root problem after some more poking at my container:

/etc/systemd/system-generators/lxc: 88: [: Illegal number: 256~rc3

If I hardcode systemd_version=256 in that script and restart the container, everything works properly again. So, it appears there's a bug in the generator script (line 170) that's always expecting to get an int without anything after it, such as "~rc3". This issue will go away after systemd is updated to a non-rc version.

I don't know if you want to re-assign this bug to distrobuilder or not, but it's not directly an apparmor issue like I first thought.

[stgraber]

Ah, interesting. The symfollow thing is still an issue which I'm addressing but indeed if the generator doesn't work at all, then that's another problem too :)

[stgraber]

lxc/distrobuilder#849