Support for Bottlerocket, EKS and Kubernetes Node Problem Detector (NPD)

[btiernay]

🔍 Context

The standard Node Problem Detector (NPD) is a key component of Kubernetes clusters that focus on minimizing service disruption through deep health monitoring of the underlying node operating system.

NPD is enabled by default in:

• Google GKE as a Kubernetes Addon (code)
• Azure AKS as part of the AKS Linux Extension (code)

This suggests that it is a mature software component that should equally be viable for AWS EKS. However, AWS EKS does not offer this out of the box.

On AWS, one might assume that installing the Node Termination Handler (NTH) would be sufficient and make NPD unnecessary. However, as we've discovered this is insufficient for many scenarios in which AWS maintenance event is not published. It turns out that NTH and NPD are complementary, with different focus areas, strengths, and weaknesses.

⚠️ Problem

When one attempts to self-manage NPD on Bottlerocket and EKS, it is a struggle to get it working correctly due to challenges like the inability to install on directly on the host (such as AKS and GKE do) and SELinux policy constraints when running as a DaemonSet. Furthermore, privileged access is required to interact with host domain sockets and related resources.

💡 Idea

This idea proposes giving standard guidance to the successful installation and management of NPD in a manner that limits the security risks associated with its need to interact with components such as systemd, journalctl and the container runtime. If there are upstream changes that are needed to make this more secure, these can be suggested and worked on.

Another possibility is a Bottlerocket package addition that can be run without the need of privileged access. This is facilitated via NPD's support for Custom Plugin Monitors. The idea would be to develop an interface that could be used to get the results of queries such as these:

• show containerd --property=InactiveExitTimestamp
• crictl --runtime-endpoint=unix:///var/run/containerd/containerd.sock pods --latest
• journalctl --unit <service> --since <time>

Thank you community for your consideration! 🙇

[bcressey]

I would expect most of these accesses to work from a privileged: true container, except for systemctl show since those queries are blocked. My inclination would be to try to ease the policy restrictions somehow so that read-only, non-mutating interactions with systemd can be permitted.

[btiernay]

Thank you for your input @bcressey 🙇 .

I would expect most of these accesses to work from a privileged: true container, except for systemctl show since those queries are blocked.

That has been my experience. It's just the systemctl show interaction that has been problematic. That said, following the principle of least privilege is highly desirable or the the other interactions generally.

My inclination would be to try to ease the policy restrictions somehow so that read-only, non-mutating interactions with systemd can be permitted.

👍. Is this feasible with SELinux policies and something that you suggest creating an project issue for?

Thanks again!