Remove privileged mode with Fusion

[bentsherman]

Close #3337

Currently includes:

• Docker
• Podman
• Wave debug command
• AWS Batch

Not sure yet what's needed for Singularity, but there is the --fusemount option.

Not sure if AWS Batch supports device capabilities for containers. The --device option is supported under LinuxParameters and a Device can have permissions READ | WRITE | MKNOD.

Waiting for @jordeu to advise on the remaining implementation.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	51d3297
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/662ab98e1364d100086b7ea0

[pditommaso]

Were these changes tested?

[bentsherman]

Not yet, just following your and Jordi's suggestions. I can test them later, and try K8s to see if any permissions are not needed.

[pditommaso]

This stuff is tricky, and there may be differences across linux and mac implementations. It could even be necessary to implement as an opt-in feature.

[bentsherman]

Regarding the testing, haven't all of these changes except for K8s already been tested? I assume you guys tested docker and podman by setting docker.fusionOptions.

[pditommaso]

I've tried once, but was a kind of mess. Also there have been a lot of changes recently with this problem with the root user, etc. Nothing can be taken for granted

[bentsherman]

@pditommaso do you still want to merge the non-K8s bits of this PR?

[pditommaso]

it would be nice, @jordeu recently wrote some useful guidelines about this

• Fuse device /dev/fuse should be mounted inside the container.
• with Kernel version below 4.18, you need to run it with CAP_SYS_ADMIN capability or --privileged flag.
• with Docker you need to unconfine the container with --security-opt apparmor=unconfined option or define an apparmor profile that allows the container to use fuse devices.
• Also seccomp profile should be unconfined with --security-opt seccomp=unconfined option in some cases.

[pditommaso]

@jordeu can you please review this one more time?

[jordeu]

I've double-checked, and everything is working

[pditommaso]

It turns out, it does not work on mac

2024-04-25 22:35:26 8:35PM WRN Error running in a new user namespace - fork/exec /usr/bin/fusion: invalid argument
2024-04-25 22:35:26 
2024-04-25 22:35:29 8:35PM ERR mount.go:204 > mount fail error="exec: \"/bin/fusermount\": stat /bin/fusermount: no such file or directory"

[jordeu]

It turns out, it does not work on mac

2024-04-25 22:35:26 8:35PM WRN Error running in a new user namespace - fork/exec /usr/bin/fusion: invalid argument
2024-04-25 22:35:26 
2024-04-25 22:35:29 8:35PM ERR mount.go:204 > mount fail error="exec: \"/bin/fusermount\": stat /bin/fusermount: no such file or directory"

With docker or podman?

[pditommaso]

Docker

[jordeu]

Are you using Docker Desktop? What version? Can you check the Kernel version (run uname -a inside a container)?

[pditommaso]

# uname -a 
Linux 380dfabbe99c 6.6.22-linuxkit #1 SMP Fri Mar 29 12:21:27 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux