Unhealthy installation with helm

[chbalbas]

Hello comunity,

I am trying to install Linkerd in a k8s cluster with helm (2.11.5 version https://artifacthub.io/packages/helm/linkerd2/linkerd2) but I am getting some errors difficult to debut it.

I have passed the linkerd check --pre checks

helm install linkerd2 \
  --set-file identityTrustAnchorsPEM=ca.crt \
  --set-file identity.issuer.tls.crtPEM=issuer.crt \
  --set-file identity.issuer.tls.keyPEM=issuer.key \
  linkerd/linkerd2 \
  -f ../values.yaml -n linkerd --create-namespace

NAME                                          READY   STATUS             RESTARTS         AGE
pod/linkerd-destination-7d6db75854-rhm2p      0/4     CrashLoopBackOff   65 (2m35s ago)   44m
pod/linkerd-identity-f694d95f-trrb8           0/2     CrashLoopBackOff   30 (2m54s ago)   44m
pod/linkerd-proxy-injector-74d69dbd54-hwsdm   0/2     CrashLoopBackOff   28 (2m5s ago)    44m

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/linkerd-dst                 ClusterIP   172.17.0.131   <none>        8086/TCP   44m
service/linkerd-dst-headless        ClusterIP   None           <none>        8086/TCP   44m
service/linkerd-identity            ClusterIP   172.17.2.254   <none>        8080/TCP   44m
service/linkerd-identity-headless   ClusterIP   None           <none>        8080/TCP   44m
service/linkerd-policy              ClusterIP   None           <none>        8090/TCP   44m
service/linkerd-policy-validator    ClusterIP   172.17.2.221   <none>        443/TCP    44m
service/linkerd-proxy-injector      ClusterIP   172.17.2.99    <none>        443/TCP    44m
service/linkerd-sp-validator        ClusterIP   172.17.1.150   <none>        443/TCP    44m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/linkerd-destination      0/1     1            0           44m
deployment.apps/linkerd-identity         0/1     1            0           44m
deployment.apps/linkerd-proxy-injector   0/1     1            0           44m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/linkerd-destination-7d6db75854      1         1         0       44m
replicaset.apps/linkerd-identity-f694d95f           1         1         0       44m
replicaset.apps/linkerd-proxy-injector-74d69dbd54   1         1         0       44m

$ k get configmap -n linkerd
NAME                           DATA   AGE
kube-root-ca.crt               1      45m
linkerd-config                 1      44m
linkerd-identity-trust-roots   1      44m

EVENTS

Linkerd-destination

$ k describe pod/linkerd-destination-7d6db75854-rhm2p
Events:
  Type     Reason               Age                    From               Message
  ----     ------               ----                   ----               -------
  Normal   Scheduled            46m                    default-scheduler  Successfully assigned linkerd/linkerd-destination-7d6db75854-rhm2p to aks-poolcipdev01-37231988-vmss00000i
  Normal   Pulled               46m                    kubelet            Container image "****/linkerd/proxy-init:v1.5.3" already present on machine
  Normal   Created              46m                    kubelet            Created container linkerd-init
  Normal   Started              46m                    kubelet            Started container linkerd-init
  Normal   Created              46m                    kubelet            Created container destination
  Normal   Created              46m                    kubelet            Created container sp-validator
  Normal   Started              46m (x2 over 46m)      kubelet            Started container linkerd-proxy
  Warning  FailedPostStartHook  46m                    kubelet            Exec lifecycle hook ([/usr/lib/linkerd/linkerd-await --timeout=2m]) for Container "linkerd-proxy" in Pod "linkerd-destination-7d6db75854-rhm2p_linkerd(587866e3-3f8c-40bf-94b6-f14c7354f7d5)" failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "4d6374d5476980d54c18365e89735a942f99cbad72da68450215cde2ea68404f": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: read init-p: connection reset by peer: unknown, message: ""
  Normal   Started              46m                    kubelet            Started container policy
  Normal   Pulled               46m                    kubelet            Container image "****/linkerd/controller:stable-2.11.5" already present on machine
  Normal   Pulled               46m (x2 over 46m)      kubelet            Container image "****/linkerd/proxy:stable-2.11.5" already present on machine
  Normal   Started              46m                    kubelet            Started container destination
  Normal   Pulled               46m                    kubelet            Container image "****/linkerd/controller:stable-2.11.5" already present on machine
  Normal   Created              46m (x2 over 46m)      kubelet            Created container linkerd-proxy
  Normal   Started              46m                    kubelet            Started container sp-validator
  Normal   Pulled               46m                    kubelet            Container image "****/linkerd/policy-controller:stable-2.11.5" already present on machine
  Normal   Created              46m                    kubelet            Created container policy
  Normal   Killing              46m (x2 over 46m)      kubelet            FailedPostStartHook
  Warning  FailedPostStartHook  46m                    kubelet            Exec lifecycle hook ([/usr/lib/linkerd/linkerd-await --timeout=2m]) for Container "linkerd-proxy" in Pod "linkerd-destination-7d6db75854-rhm2p_linkerd(587866e3-3f8c-40bf-94b6-f14c7354f7d5)" failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "64c52e4b38f6b5914fedc0e51589caa52e89dfedae217be501ea5303efb2120d": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "/usr/lib/linkerd/linkerd-await": stat /usr/lib/linkerd/linkerd-await: no such file or directory: unknown, message: ""
  Warning  Unhealthy            46m                    kubelet            Readiness probe failed: Get "http://172.16.7.88:9996/ready": dial tcp 172.16.7.88:9996: connect: connection refused
  Warning  BackOff              11m (x282 over 46m)    kubelet            Back-off restarting failed container
  Warning  Unhealthy            6m26s (x199 over 46m)  kubelet            Readiness probe failed: Get "http://172.16.7.88:9997/ready": dial tcp 172.16.7.88:9997: connect: connection refused
  Warning  Unhealthy            83s (x110 over 46m)    kubelet            Readiness probe failed: Get "http://172.16.7.88:9990/ready": dial tcp 172.16.7.88:9990: connect: connection refused

Linkerd identity

$ k describe pod linkerd-identity-f694d95f-trrb8

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  49m                    default-scheduler  Successfully assigned linkerd/linkerd-identity-f694d95f-trrb8 to aks-poolcipdev01-37231988-vmss00000j
  Normal   Started    49m                    kubelet            Started container identity
  Normal   Created    49m                    kubelet            Created container linkerd-init
  Normal   Started    49m                    kubelet            Started container linkerd-init
  Normal   Created    49m                    kubelet            Created container identity
  Normal   Pulled     49m                    kubelet            Container image "****/linkerd/proxy-init:v1.5.3" already present on machine
  Normal   Created    49m (x2 over 49m)      kubelet            Created container linkerd-proxy
  Normal   Started    49m (x2 over 49m)      kubelet            Started container linkerd-proxy
  Warning  BackOff    49m (x4 over 49m)      kubelet            Back-off restarting failed container
  Warning  Unhealthy  49m                    kubelet            Liveness probe failed: Get "http://172.16.3.45:9990/ping": dial tcp 172.16.3.45:9990: connect: connection refused
  Normal   Pulled     49m (x3 over 49m)      kubelet            Container image "****/linkerd/proxy:stable-2.11.5" already present on machine
  Warning  Unhealthy  34m (x98 over 49m)     kubelet            Readiness probe failed: Get "http://172.16.3.45:9990/ready": dial tcp 172.16.3.45:9990: connect: connection refused
  Normal   Pulled     9m25s (x17 over 49m)   kubelet            Container image "****/linkerd/controller:stable-2.11.5" already present on machine
  Warning  BackOff    4m17s (x181 over 45m)  kubelet            Back-off restarting failed container

Linkerd proxy injector

$ k describe pod linkerd-proxy-injector-74d69dbd54-hwsdm

Events:
  Type     Reason               Age                  From               Message
  ----     ------               ----                 ----               -------
  Normal   Scheduled            51m                  default-scheduler  Successfully assigned linkerd/linkerd-proxy-injector-74d69dbd54-hwsdm to aks-poolcipdev01-37231988-vmss00000f
  Normal   Pulled               51m                  kubelet            Container image "****/linkerd/proxy-init:v1.5.3" already present on machine
  Normal   Created              51m                  kubelet            Created container linkerd-init
  Normal   Started              51m                  kubelet            Started container linkerd-init
  Normal   Pulled               51m                  kubelet            Container image "****/linkerd/controller:stable-2.11.5" already present on machine
  Warning  FailedPostStartHook  51m                  kubelet            Exec lifecycle hook ([/usr/lib/linkerd/linkerd-await --timeout=2m]) for Container "linkerd-proxy" in Pod "linkerd-proxy-injector-74d69dbd54-hwsdm_linkerd(85d0cc6b-116d-4e7c-b4bd-ed8dd2a58b27)" failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "f1d98ad55b6ce673f236fc3047658ae8a143376bbd7c1d0cdffa7b69377608eb": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "/usr/lib/linkerd/linkerd-await": stat /usr/lib/linkerd/linkerd-await: no such file or directory: unknown, message: ""
  Normal   Created              51m                  kubelet            Created container proxy-injector
  Normal   Started              51m                  kubelet            Started container proxy-injector
  Normal   Created              51m (x2 over 51m)    kubelet            Created container linkerd-proxy
  Normal   Started              51m (x2 over 51m)    kubelet            Started container linkerd-proxy
  Normal   Killing              51m (x2 over 51m)    kubelet            FailedPostStartHook
  Normal   Pulled               51m (x2 over 51m)    kubelet            Container image "****/linkerd/proxy:stable-2.11.5" already present on machine
  Warning  FailedPostStartHook  51m                  kubelet            Exec lifecycle hook ([/usr/lib/linkerd/linkerd-await --timeout=2m]) for Container "linkerd-proxy" in Pod "linkerd-proxy-injector-74d69dbd54-hwsdm_linkerd(85d0cc6b-116d-4e7c-b4bd-ed8dd2a58b27)" failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "7156fbdc52ae66952dd4e6e825652b1c345ee53d047547adafe15ae4d3986708": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: read init-p: connection reset by peer: unknown, message: ""
  Warning  Unhealthy            11m (x189 over 51m)  kubelet            Readiness probe failed: Get "http://172.16.6.66:9995/ready": dial tcp 172.16.6.66:9995: connect: connection refused
  Warning  BackOff              74s (x276 over 51m)  kubelet            Back-off restarting failed container 

Logs

Linkerd destination pod

linkerd proxy container

$ k logs linkerd-destination-7d6db75854-rhm2p -c linkerd-proxy
Error: --server-tls-key must be set 🔴

linkerd destination container

$ k logs linkerd-destination-7d6db75854-rhm2p -c destination
time="2023-05-11T09:32:53Z" level=info msg="running version stable-2.11.5"
time="2023-05-11T09:32:53Z" level=info msg="Using default opaque ports: map[25:{} 587:{} 3306:{} 4444:{} 5432:{} 6379:{} 9300:{} 11211:{}]"
time="2023-05-11T09:32:53Z" level=info msg="waiting for caches to sync"
time="2023-05-11T09:32:53Z" level=info msg="caches synced"
time="2023-05-11T09:32:53Z" level=info msg="starting admin server on :9996"
time="2023-05-11T09:32:53Z" level=info msg="starting gRPC server on :8086"
time="2023-05-11T09:33:34Z" level=info msg="shutting down gRPC server on :8086" 🔴

linkerd destination sp-validator container

$ k logs linkerd-destination-7d6db75854-rhm2p -c sp-validator
time="2023-05-11T09:32:53Z" level=info msg="running version stable-2.11.5"
time="2023-05-11T09:32:53Z" level=info msg="waiting for caches to sync"
time="2023-05-11T09:32:53Z" level=info msg="caches synced"
time="2023-05-11T09:32:53Z" level=info msg="starting admin server on :9997"
time="2023-05-11T09:32:53Z" level=info msg="listening at :8443"
time="2023-05-11T09:33:32Z" level=info msg="shutting down webhook server" 🔴

linkerd destination policy container

$ k logs linkerd-destination-7d6db75854-rhm2p -c policy
2023-05-11T09:35:56.948371Z  INFO grpc{port=8090}: linkerd_policy_controller: gRPC server listening addr=0.0.0.0:8090

Linkerd destination pod

linkerd destination identity container

$ k logs linkerd-identity-f694d95f-trrb8 -c identity
time="2023-05-11T09:33:13Z" level=info msg="running version stable-2.11.5"
time="2023-05-11T09:33:13Z" level=info msg="starting admin server on :9990"
time="2023-05-11T09:33:13Z" level=info msg="starting gRPC server on :8080"
time="2023-05-11T09:33:52Z" level=info msg="shutting down gRPC server on :8080

linkerd destination linkerdy-proxy container

$ k logs linkerd-identity-f694d95f-trrb8 -c linkerd-proxy
Error: --server-tls-key must be set 🔴

Linkerd proxy-injector pod

linkerd proxy injector - linkerd proxy container

$ k logs linkerd-proxy-injector-74d69dbd54-hwsdm -c linkerd-proxy
Error: --server-tls-key must be set 🔴

linkerd proxy injector - proxy-injector container

$ k logs linkerd-proxy-injector-74d69dbd54-hwsdm -c proxy-injector
time="2023-05-11T09:40:49Z" level=info msg="running version stable-2.11.5"
time="2023-05-11T09:40:49Z" level=info msg="waiting for caches to sync"
time="2023-05-11T09:40:49Z" level=info msg="listening at :8443"

Linkerd check

$ linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
/ pod/linkerd-destination-7d6db75854-rhm2p container sp-validator is not ready 🔴 

Issue

Error: --server-tls-key must be set I don't know how to solve this error, the container yaml spec does not have any related with --server-tls-key. Any ideas? Thanks for helping 😄

[adleong]

Hi @chbalbas

The Linkerd proxy should not be expecting a --server-tls-key flag, since this flag only applies to the policy controller. Is it possible that you are accidentally using the policy controller docker image as your proxy image? I see that you are supplying a values.yaml file. Can you share the contents of that file?

[chbalbas]

You was right! The problem was that I have pushed wrongly the images...

Thank you very much

[alanjino]

can you help me i am facing this isssue ?