control-plane pods stuck in PodInitializing

[ccmtaylor]

Hi all,

we're currently evaluating service mesh products for use in our (on-prem/bare-metal and AWS) Kubernetes clusters. After checking that linkerd check --pre succeeded, I tried installing linkerd on one of our clusters, using linkerd install --ha | kubectl apply -f -. Unfortunately, the linkerd-destination and linkerd-proxy-injector pods get stuck in state PodInitializing indefinitely (I waited ~3h). The init-container completes successfully, and its logs look reasonable, but the regular containers never start up.

This looks similar to #7271, except in my case, the pods don't get out of the PodInitializing state at all. The cluster is not low on resources, it's a ~50 node cluster with very few workloads.

I'd appreciate any pointers to things I may have overlooked.

Thanks,
Chris

linkerd/k8s versions

$ kubectl version --short
Client Version: v1.21.5
Server Version: v1.20.11

$ linkerd version
Client version: stable-2.11.1
Server version: stable-2.11.1

kubectl -n linkerd get all

$ kubectl -n linkerd get all
NAME                                         READY   STATUS            RESTARTS   AGE
pod/linkerd-destination-795f6956b9-cvfp7     0/4     PodInitializing   0          25s
pod/linkerd-destination-795f6956b9-dnv62     0/4     PodInitializing   0          25s
pod/linkerd-destination-795f6956b9-mnf22     0/4     PodInitializing   0          25s
pod/linkerd-identity-684689b69f-ctptb        2/2     Running           0          27s
pod/linkerd-identity-684689b69f-stm4l        2/2     Running           0          27s
pod/linkerd-identity-684689b69f-tk9t7        2/2     Running           0          27s
pod/linkerd-proxy-injector-b5fcd7c9f-858jd   0/2     PodInitializing   0          24s
pod/linkerd-proxy-injector-b5fcd7c9f-dckws   0/2     PodInitializing   0          24s
pod/linkerd-proxy-injector-b5fcd7c9f-znjrf   0/2     PodInitializing   0          24s

NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/linkerd-dst                 ClusterIP   10.126.137.188   <none>        8086/TCP   28s
service/linkerd-dst-headless        ClusterIP   None             <none>        8086/TCP   27s
service/linkerd-identity            ClusterIP   10.126.37.138    <none>        8080/TCP   29s
service/linkerd-identity-headless   ClusterIP   None             <none>        8080/TCP   29s
service/linkerd-policy              ClusterIP   None             <none>        8090/TCP   27s
service/linkerd-policy-validator    ClusterIP   10.126.81.11     <none>        443/TCP    26s
service/linkerd-proxy-injector      ClusterIP   10.126.1.192     <none>        443/TCP    25s
service/linkerd-sp-validator        ClusterIP   10.126.234.166   <none>        443/TCP    27s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/linkerd-destination      0/3     3            0           26s
deployment.apps/linkerd-identity         3/3     3            3           28s
deployment.apps/linkerd-proxy-injector   0/3     3            0           25s

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/linkerd-destination-795f6956b9     3         3         0       26s
replicaset.apps/linkerd-identity-684689b69f        3         3         3       28s
replicaset.apps/linkerd-proxy-injector-b5fcd7c9f   3         3         0       25s

NAME                              SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/linkerd-heartbeat   55 11 * * *   False     0        <none>          26s

kubectl describe pod linkerd-destination

Name:                 linkerd-destination-7b77fffcf6-lp7mp
Namespace:            linkerd
Priority:             1000
Priority Class Name:  default
Node:                 some-node-name/10.33.89.55
Start Time:           Wed, 15 Dec 2021 15:31:39 +0100
Labels:               linkerd.io/control-plane-component=destination
                      linkerd.io/control-plane-ns=linkerd
                      linkerd.io/proxy-deployment=linkerd-destination
                      linkerd.io/workload-ns=linkerd
                      pod-template-hash=7b77fffcf6
Annotations:          checksum/config: ae27873051d6ab5b7d13fd0fb127c244d8b5d11c25996c46862be1b908e626f9
                      config.linkerd.io/default-inbound-policy: all-unauthenticated
                      linkerd.io/created-by: linkerd/cli stable-2.11.1
                      linkerd.io/identity-mode: default
                      linkerd.io/proxy-version: stable-2.11.1
Status:               Pending
IP:                   10.224.55.41
IPs:
  IP:           10.224.55.41
Controlled By:  ReplicaSet/linkerd-destination-7b77fffcf6
Init Containers:
  linkerd-init:
    Container ID:  docker://d68b9c6908862d1330d05e03f7a4244581c29cfba7b3c5b0a3d6ad68ae1fd1fd
    Image:         cr.l5d.io/linkerd/proxy-init:v1.4.0
    Image ID:      docker-pullable://cr.l5d.io/linkerd/proxy-init@sha256:60d12fbb0b4a53962a5c2a59b496b3ee20052d26c0c56fd2ee38fd7fae62146e
    Port:          <none>
    Host Port:     <none>
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191,4567,4568
      --outbound-ports-to-ignore
      443
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 15 Dec 2021 15:31:40 +0100
      Finished:     Wed, 15 Dec 2021 15:31:40 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:        10m
      memory:     10Mi
    Environment:  <none>
    Mounts:
      /run from linkerd-proxy-init-xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from linkerd-destination-token-47vnd (ro)
Containers:
  linkerd-proxy:
    Container ID:   
    Image:          cr.l5d.io/linkerd/proxy:stable-2.11.1
    Image ID:       
    Ports:          4143/TCP, 4191/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:4191/live delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:4191/ready delay=2s timeout=1s period=10s #success=1 #failure=3
    Environment:
      _pod_name:                                                linkerd-destination-7b77fffcf6-lp7mp (v1:metadata.name)
      _pod_ns:                                                  linkerd (v1:metadata.namespace)
      _pod_nodeName:                                             (v1:spec.nodeName)
      LINKERD2_PROXY_LOG:                                       warn,linkerd=info
      LINKERD2_PROXY_LOG_FORMAT:                                plain
      LINKERD2_PROXY_DESTINATION_SVC_ADDR:                      localhost.:8086
      LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS:              10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_POLICY_SVC_ADDR:                           localhost.:8090
      LINKERD2_PROXY_POLICY_WORKLOAD:                           $(_pod_ns):$(_pod_name)
      LINKERD2_PROXY_INBOUND_DEFAULT_POLICY:                    all-unauthenticated
      LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS:                   10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT:                   100ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT:                  1000ms
      LINKERD2_PROXY_CONTROL_LISTEN_ADDR:                       0.0.0.0:4190
      LINKERD2_PROXY_ADMIN_LISTEN_ADDR:                         0.0.0.0:4191
      LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR:                      127.0.0.1:4140
      LINKERD2_PROXY_INBOUND_LISTEN_ADDR:                       0.0.0.0:4143
      LINKERD2_PROXY_INBOUND_IPS:                                (v1:status.podIPs)
      LINKERD2_PROXY_INBOUND_PORTS:                             8086,8090,8443,9443,9990,9996,9997
      LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES:              svc.cluster.local.
      LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE:                  10000ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE:                10000ms
      LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION:  25,587,3306,4444,5432,6379,9300,11211
      LINKERD2_PROXY_DESTINATION_CONTEXT:                       {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)"}
                                                                
      _pod_sa:                                                   (v1:spec.serviceAccountName)
      _l5d_ns:                                                  linkerd
      _l5d_trustdomain:                                         cluster.local
      LINKERD2_PROXY_IDENTITY_DIR:                              /var/run/linkerd/identity/end-entity
      LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS:                    <set to the key 'ca-bundle.crt' of config map 'linkerd-identity-trust-roots'>  Optional: false
      LINKERD2_PROXY_IDENTITY_TOKEN_FILE:                       /var/run/secrets/kubernetes.io/serviceaccount/token
      LINKERD2_PROXY_IDENTITY_SVC_ADDR:                         linkerd-identity-headless.linkerd.svc.cluster.local.:8080
      LINKERD2_PROXY_IDENTITY_LOCAL_NAME:                       $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_IDENTITY_SVC_NAME:                         linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_DESTINATION_SVC_NAME:                      linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_POLICY_SVC_NAME:                           linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from linkerd-destination-token-47vnd (ro)
  destination:
    Container ID:  
    Image:         cr.l5d.io/linkerd/controller:stable-2.11.1
    Image ID:      
    Ports:         8086/TCP, 9996/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      destination
      -addr=:8086
      -controller-namespace=linkerd
      -enable-h2-upgrade=true
      -log-level=info
      -log-format=plain
      -enable-endpoint-slices=false
      -cluster-domain=cluster.local
      -identity-trust-domain=cluster.local
      -default-opaque-ports=25,587,3306,4444,5432,6379,9300,11211
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:9996/ping delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9996/ready delay=0s timeout=1s period=10s #success=1 #failure=7
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from linkerd-destination-token-47vnd (ro)
  sp-validator:
    Container ID:  
    Image:         cr.l5d.io/linkerd/controller:stable-2.11.1
    Image ID:      
    Ports:         8443/TCP, 9997/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      sp-validator
      -log-level=info
      -log-format=plain
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:9997/ping delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9997/ready delay=0s timeout=1s period=10s #success=1 #failure=7
    Environment:    <none>
    Mounts:
      /var/run/linkerd/tls from sp-tls (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from linkerd-destination-token-47vnd (ro)
  policy:
    Container ID:  
    Image:         cr.l5d.io/linkerd/policy-controller:stable-2.11.1
    Image ID:      
    Ports:         8090/TCP, 9990/TCP, 9443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      --admin-addr=0.0.0.0:9990
      --grpc-addr=0.0.0.0:8090
      --admission-addr=0.0.0.0:9443
      --cluster-networks=10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      --identity-domain=cluster.local
      --default-policy=all-unauthenticated
      --log-level=linkerd=info,warn
      --log-format=plain
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:admin/ready delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:admin/ready delay=0s timeout=1s period=10s #success=1 #failure=7
    Environment:    <none>
    Mounts:
      /var/run/linkerd/tls from policy-tls (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from linkerd-destination-token-47vnd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  sp-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  linkerd-sp-validator-k8s-tls
    Optional:    false
  policy-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  linkerd-policy-validator-k8s-tls
    Optional:    false
  linkerd-proxy-init-xtables-lock:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  linkerd-identity-end-entity:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  linkerd-destination-token-47vnd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  linkerd-destination-token-47vnd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  28m   default-scheduler  Successfully assigned linkerd/linkerd-destination-7b77fffcf6-lp7mp to some-node-name
  Normal  Pulled     28m   kubelet            Container image "cr.l5d.io/linkerd/proxy-init:v1.4.0" already present on machine
  Normal  Created    28m   kubelet            Created container linkerd-init
  Normal  Started    28m   kubelet            Started container linkerd-init
  Normal  Pulled     28m   kubelet            Container image "cr.l5d.io/linkerd/proxy:stable-2.11.1" already present on machine
  Normal  Created    28m   kubelet            Created container linkerd-proxy
  Normal  Started    28m   kubelet            Started container linkerd-proxy

linkerd-init logs

2021/12/15 14:31:40 Tracing this script execution as [1639578700]
2021/12/15 14:31:40 current state
------------------------------------------------------------
2021/12/15 14:31:40 :; iptables-save
2021/12/15 14:31:40 

2021/12/15 14:31:40 configuration
------------------------------------------------------------
2021/12/15 14:31:40 Will ignore port [4190 4191 4567 4568] on chain PROXY_INIT_REDIRECT
2021/12/15 14:31:40 Will redirect all INPUT ports to proxy
2021/12/15 14:31:40 Ignoring uid 2102
2021/12/15 14:31:40 Will ignore port [443] on chain PROXY_INIT_OUTPUT
2021/12/15 14:31:40 Redirecting all OUTPUT to 4140
2021/12/15 14:31:40 

2021/12/15 14:31:40 adding rules
------------------------------------------------------------
2021/12/15 14:31:40 :; iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191,4567,4568 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191,4567,4568/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1639578700
2021/12/15 14:31:40 :; iptables -t nat -N PROXY_INIT_OUTPUT -m comment --comment proxy-init/redirect-common-chain/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -j RETURN -m comment --comment proxy-init/ignore-proxy-user-id/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN -m comment --comment proxy-init/ignore-loopback/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A PROXY_INIT_OUTPUT -p tcp --match multiport --dports 443 -j RETURN -m comment --comment proxy-init/ignore-port-443/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 4140 -m comment --comment proxy-init/redirect-all-outgoing-to-proxy-port/1639578700
2021/12/15 14:31:40 :; iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1639578700
2021/12/15 14:31:40 

2021/12/15 14:31:40 end state
------------------------------------------------------------
2021/12/15 14:31:40 :; iptables-save
2021/12/15 14:31:40 # Generated by iptables-save v1.8.7 on Wed Dec 15 14:31:40 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:PROXY_INIT_OUTPUT - [0:0]
:PROXY_INIT_REDIRECT - [0:0]
-A PREROUTING -m comment --comment "proxy-init/install-proxy-init-prerouting/1639578700" -j PROXY_INIT_REDIRECT
-A OUTPUT -m comment --comment "proxy-init/install-proxy-init-output/1639578700" -j PROXY_INIT_OUTPUT
-A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -m comment --comment "proxy-init/ignore-proxy-user-id/1639578700" -j RETURN
-A PROXY_INIT_OUTPUT -o lo -m comment --comment "proxy-init/ignore-loopback/1639578700" -j RETURN
-A PROXY_INIT_OUTPUT -p tcp -m multiport --dports 443 -m comment --comment "proxy-init/ignore-port-443/1639578700" -j RETURN
-A PROXY_INIT_OUTPUT -p tcp -m comment --comment "proxy-init/redirect-all-outgoing-to-proxy-port/1639578700" -j REDIRECT --to-ports 4140
-A PROXY_INIT_REDIRECT -p tcp -m multiport --dports 4190,4191,4567,4568 -m comment --comment "proxy-init/ignore-port-4190,4191,4567,4568/1639578700" -j RETURN
-A PROXY_INIT_REDIRECT -p tcp -m comment --comment "proxy-init/redirect-all-incoming-to-proxy-port/1639578700" -j REDIRECT --to-ports 4143
COMMIT
# Completed on Wed Dec 15 14:31:40 2021

2021/12/15 14:31:40 

[mateiidavid]

Hey @ccmtaylor, according to the kubectl describe, I see that the proxy container started. Does it log anything useful? We block on proxy readiness through postStart hooks by default. I'd assume if the proxy can't start then the main container can't start either. If the logs are being a bit vague on the proxy, you can try to bump the level to linkerd=debug,warn

[ccmtaylor]

I took another look. @olix0r, your hunch about DNS was correct. After manually deleting the postStart hooks from the linkerd install output, the proxy logs contained many messages about not being able to resolve the identity service.

In our setup, we use a different, non-default, cluster domain for each cluster. Re-installing linkerd with --cluster-domain=our.domain (and linkerd-viz with --set clusterDomain=our.domain) made everything work.

It's unfortunate that there is no easy way to debug this via e.g. logs. kubernetes/kubernetes#30042 suggested making the output of lifecycle hooks visible, but the issue didn't go anywhere. Would this be something that linkerd check could detect?

Alternatively, linkerd could use relative hostnames (i.e. without the .cluster.local suffix). However, I suspect, it may cause issues with mTLS because host names don't match up with certificates anymore?

[olix0r]

@ccmtaylor Excellent. Glad you sorted this out.

Would this be something that linkerd check could detect?

I'm not sure of any way to detect the cluster domain from the API server; so I think we'd have to actually run a Job in the cluster to exercise the cluster's DNS to refine a relative domain to a canonical domain. While this isn't necessarily a bad idea, it's out of scope of what check currently does.

Alternatively, linkerd could use relative hostnames (i.e. without the .cluster.local suffix). However, I suspect, it may cause issues with mTLS because host names don't match up with certificates anymore?

Linkerd's mTLS isn't actually tied to DNS, so this could potentially work. There's a separate "trust domain" configuration, however, that is used to differentiate the certificate spaces between clusters/orgs. We chose to use fully-qualified names (i.e. with a trailing dot) to completely avoid DNS search path resolution, which, if I recall correctly, could cause problems in some environments (especially if non-default search paths were configured on a pod). I lean towards using explicit names to avoid ambiguity, but we really should figure out a better way to surface this misconfiguration.

At the very least, this sounds like something we should include in the troubleshooting guide.

[olix0r]

Unfortunately kubernetes/website#25758 recently staled out; but this proposed an API for discovering the cluster domain. If such an API existed, check could use it.

[jingb]

I meet the same problems.
I install linkerd-viz by Helm command.
helm install linkerd-viz -n linkerd-viz --set clusterDomain={{my custom cluster domain}} linkerd/linkerd-viz
the pods under linkerd-viz stuck PodInitializing about serveral minutes, but all will get ready finally.
And I didn't modify anything.
I have install linkerd by helm successfully and set the correct clusterdomain before.

kubectl get pod -n linkerd-viz
NAME READY STATUS RESTARTS AGE
grafana-8d54d5f6d-vvzpv 0/2 PodInitializing 0 34m
metrics-api-dbb4877d-jrf7c 2/2 Running 0 34m
prometheus-7bbc4d8c5b-csbj5 2/2 Running 0 34m
tap-7b47666659-fqtvj 2/2 Running 0 34m
tap-injector-58f7677d96-5npks 2/2 Running 0 34m
web-6966b48798-w6q5s 0/2 PodInitializing 0 5m21s
web-6c6b4f7d5f-22j4b 2/2 Running 0 34m

[ksepulvedag]

I meet the same problems. I install linkerd-viz by Helm command. helm install linkerd-viz -n linkerd-viz --set clusterDomain={{my custom cluster domain}} linkerd/linkerd-viz the pods under linkerd-viz stuck PodInitializing about serveral minutes, but all will get ready finally. And I didn't modify anything. I have install linkerd by helm successfully and set the correct clusterdomain before.

kubectl get pod -n linkerd-viz NAME READY STATUS RESTARTS AGE grafana-8d54d5f6d-vvzpv 0/2 PodInitializing 0 34m metrics-api-dbb4877d-jrf7c 2/2 Running 0 34m prometheus-7bbc4d8c5b-csbj5 2/2 Running 0 34m tap-7b47666659-fqtvj 2/2 Running 0 34m tap-injector-58f7677d96-5npks 2/2 Running 0 34m web-6966b48798-w6q5s 0/2 PodInitializing 0 5m21s web-6c6b4f7d5f-22j4b 2/2 Running 0 34m

Thank you, this one worked for me too, my cluster name isn't the default name...