"No endpoints" and "failfast" issues during deployments

[bjoernw]

I have two services deployed. Service A and Service B, both meshed. Service A federates to Service B. During a rolling deployment for service B I see the following logs in Service A's linkerd proxy:

INFO ThreadId(02) outbound:proxy{addr=172.20.46.68:80}: linkerd_proxy_api_resolve::resolve: No endpoints
...
INFO ThreadId(03) outbound:proxy{addr=172.20.46.68:80}:rescue{client.addr=10.30.93.75:54054}: linkerd_app_core::errors::respond: Request failed error=HTTP Balancer service in fail-fast
...
WARN ThreadId(02) outbound:proxy{addr=172.20.46.68:80}: linkerd_stack::failfast: HTTP Logical entering failfast after 3s

along with a significant, if not complete drop in QPS on service B. This happens in 1/10 deployments and tends to happen about 1min into the deployment. Unfortunately, this pretty much causes a complete outage for 30seconds until it recovers by itself.

Service A has about 20 pods and Service B 120 pods. Prior to 2.12.4 I thought maybe this had to do with the bug that was fixed in 2.12.4 around stale endpoints when using EndpointSlice objects since that seemed to make sense if linkerd was still attempting to route traffic to shut down pods due to a stale cache. But I am now on 2.12.4 and am still experiencing this issue.

Any help identifying the root cause here would be greatly appreciated.

P.S.
This does not happen on services where we use canary rollouts that cause pod counts to double and TrafficSplits slowly move traffic between stable and canary pods. This only happens on services that use canary deployments with argo-rollout's dynamic scale property that shuts down stable pods as canary pods become ready, and also on standard k8s rolling deployments.

Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2023-02-15T17:35:46Z
    see https://linkerd.io/2.12/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 2.12.3 but the latest stable version is 2.12.4
    see https://linkerd.io/2.12/checks/#l5d-version-cli for hints

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
‼ control plane and cli versions match
    control plane running stable-2.12.4 but cli running stable-2.12.3
    see https://linkerd.io/2.12/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * metrics-api-94455cc9c-97b9h (stable-2.12.3)
        * smi-adaptor-54c944956d-vxgkw (stable-2.12.3)
        * tap-554d6fdf75-q9t2h (stable-2.12.3)
        * tap-injector-569b7459c7-md5sh (stable-2.12.3)
    see https://linkerd.io/2.12/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    linkerd-destination-7d8b6587f7-2pb4f running stable-2.12.4 but cli running stable-2.12.3
    see https://linkerd.io/2.12/checks/#l5d-cp-proxy-cli-version for hints

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
    kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
    see https://linkerd.io/2.12/checks/#l5d-injection-disabled for hints

[kleimkuhler]

Just curious are you able to experiment with churning through available Pods without Argo in-between? I'd be curious if something about how it's scaling available Endpoints down is leading to staleness in the cluster in general, and not something related to Linkerd.

There are no Linkerd issues that jump to mind that may be causing this, and if it was an actual issue I feel like we would have seen more reports about this.

[jeremychase]

@bjoernw We believe this was fixed in the edge-23.2.3. Would it be possible for you to give that a try?

[bjoernw]

Sorry, wasn't receiving notifications for this discussion. That change around EndpointSlice processing (95a3d19) looks promising. @jeremychase is that going to make it into 2.13?

[mateiidavid]

@bjoernw yup! It'll be in the upcoming edge.