podman, kubernetes and more fight over the CNI config directory

[fasaxc]

I'm not sure what the solution is but as more and more apps start to use CNI, I'm seeing them fight over the /etc/cni/net.d directory. For example, RH CoreOS pre-installs podman, which includes its own CNI in /etc/cni/net.d.

If you then install kubernetes on the same system, you'd typically want to install a kubernetes focussed CNI (such as Calico or Flannel) but:

• In the window before the k8s CNI is installed, podman's CNI will get used for early-scheduled pods (ugh!)
• After the k8s CNI is installed then podman will try to use it, which is generally unexpected and likely broken.

CRIO also installs its own CNI by default.

In general the problems this causes are strange and confusing for an end user who didn't even know they had podman installed. They just see broken networking with some pods getting IPs from a different IP range.

I think it'd be good if the CNI spec could address this somehow. In general, I think it's more common to want a different CNI for each app that's installed rather than to share.

I suppose one option would be to add something to the netconf that says "this config should only be used by kubernetes/podman/whatever". Then the bootstrap CNIs installed by those apps could include such a stanza.

[mccv1r0]

I think it'd be good if the CNI spec could address this somehow.

The user of CNI could easily specify where their configuration files are located at by setting the NETCONFPATH environment variable. The binaries themselves can co-exist by each setting CNI_PATH to where they placed their binaries, either the reference binaries from here or their own.

[haircommander]

CRI-O and podman support filtering for a network name. I don't think CRI-O will use podman's network configs, and vice versa. We have a similar issue we're looking to fix in 1.24 about not installing a network config by default in the CRI-O installation so other networking providers can be the only one installed (beginning at runtime)

[haircommander]

I was wrong, CRI-O does support filtering, but doesn't do so OOTB. If you know the network config name ahead of time you can update your cri-o config to filter for that by setting cni_default_network to the network name

[danwinship]

In general, I think it's more common to want a different CNI for each app that's installed rather than to share.

Yes, exactly this. The CNI spec imagines a world where people install a random container runtime and a random CNI plugin onto a host, and the user wants the runtime to autodetect the plugin and use it. But in my experience, that's not the way that CNI gets used most of the time.

I think the solution here is that /etc/cni/net.d should just go away. CNI configurations should not be something that a host has, they should be something that a container runtime has.

[s1061123]

Unfortunately, current podman depreates CNI and CNI will be removed in next major version podman, hence it may no longer issue.

See https://blog.podman.io/2023/11/cni-deprecation-and-removal-from-podman-5-0/