Unrecoverable Node

[cyrus-mc]

I am experiencing issues where a node goes offline (Kubelet NotReady). This cluster and nodes are part of our CI/CD system which is used to run self-hosted GitHub runners. Troubleshooting is complicated by the fact that when the node gets into this state, I am unable to connect to it via SSM. AWS cloudwatch metrics show that the CPU is pegged at 100%, and the last logs messages that make it out of the system and to our log aggregation system are:

E1116 00:37:09.969498    1704 remote_runtime.go:633] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: i/o timeout\""

E1116 00:34:11.975758    1704 kubelet.go:2530] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="4h25m56.461s"

W1116 00:31:53.986619    1704 logging.go:59] [core] [Channel #4 SubChannel #5] grpc: addrConn.createTransport failed to connect to {Addr: "/run/containerd/containerd.sock", ServerName: "%2Frun%2Fcontainerd%2Fcontainerd.sock", }. Err: connection error: desc = "transport: failed to write client preface: write unix @->/run/containerd/containerd.sock: use of closed network connection"

Given I am unable to connect to the machine in anyway I am not sure how to go about discovering what is causing the instance to die.

[koooosh]

Hi @cyrus-mc , thanks for raising this concern. This seems better suited as a Github issue rather than a discussion. Can you please open an issue with the following information:

• Bottlerocket version / variant used
• Steps to reproduce error

With regard to not being able to connect to the node to see logs, you can try using the ec2::get-console-output API, like:

aws ec2 get-console-output --instance-id [INSTANCE_ID] --latest --output=text | sed 's/\\r\\n/\n/g'

[ytsssun]

Do you also have the other host metrics like memory consumption? Trying to see if this is relevant issue #4075

[cyrus-mc]

Closing and continuing discussion in #4075.