auto-injection is ignored

[jeusdi]

I've installed linkerd correctly (linkerd check --proxy -n linkerd checkings are all ok):

$ linkerd check --proxy -n linkerd
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
‼ can determine the latest version
    Get https://versioncheck.linkerd.io/version.json?version=stable-2.7.1&uuid=991db911-da8b-45c7-98b5-eb63e6162e8d&source=cli: Proxy Authentication Required
    see https://linkerd.io/checks/#l5d-version-latest for hints
‼ cli is up-to-date
    unsupported version channel: stable-2.7.1
    see https://linkerd.io/checks/#l5d-version-cli for hints

linkerd-data-plane
------------------
√ data plane namespace exists
√ data plane proxies are ready
√ data plane proxy metrics are present in Prometheus
‼ data plane is up-to-date
    linkerd/linkerd-prometheus-796d54cd7d-vr8bh: unsupported version channel: stable-2.7.1
    see https://linkerd.io/checks/#l5d-data-plane-version for hints
√ data plane and cli versions match

Status check results are √

After that, I've annotated my covid namespace with "auto-injection":

$ kubectl annotate namespace covid linkerd.io/inject=enabled

After having deployed my deployment:

$ linkerd stat deployments -n covid
NAME                MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
dev-covid-backend      0/1         -     -             -             -             -          -

$ linkerd stat pods -n covid
NAME                                 STATUS   MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
dev-covid-backend-7ccc987d4-494lv   Running      0/1         -     -             -             -             -          -

As you can see, deployment is not meshed.

I've installed linkerd correctly (linkerd check --proxy -n linkerd checkings are all ok).

After that, I've annotated my covid namespace with "auto-injection":

$ kubectl annotate namespace covid linkerd.io/inject=enabled

After having deployed my deployment:

$ linkerd stat deployments -n covid
NAME                MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
dev-covid-backend      0/1         -     -             -             -             -          -

$ linkerd stat pods -n covid
NAME                                 STATUS   MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
dev-covid-backend-7ccc987d4-494lv   Running      0/1         -     -             -             -             -          -

As you can see, deployment is not meshed.

I've trigerred heartbeat manually. I'm getting:

time="2020-05-05T12:29:39Z" level=info msg="running version stable-2.7.1"
time="2020-05-05T12:29:39Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2020-05-05T12:29:39Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2020-05-05T12:29:39Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2020-05-05T12:29:39Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2020-05-05T12:29:39Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2020-05-05T12:29:39Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2020-05-05T12:29:39Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2020-05-05T12:29:39Z" level=info msg="Sending heartbeat: https://versioncheck.linkerd.io/version.json?install-time=1588663782&k8s-version=v1.17.3%2Bk3s1&meshed-pods=9&p99-handle-us=50000&source=heartbeat&total-rps=3&uuid=991db911-da8b-45c7-98b5-eb63e6162e8d&version=stable-2.7.1"
time="2020-05-05T12:29:43Z" level=fatal msg="Failed to send heartbeat: Check URL [https://versioncheck.linkerd.io/version.json?install-time=1588663782&k8s-version=v1.17.3%2Bk3s1&meshed-pods=9&p99-handle-us=50000&source=heartbeat&total-rps=3&uuid=991db911-da8b-45c7-98b5-eb63e6162e8d&version=stable-2.7.1] request failed with: Get https://versioncheck.linkerd.io/version.json?install-time=1588663782&k8s-version=v1.17.3%2Bk3s1&meshed-pods=9&p99-handle-us=50000&source=heartbeat&total-rps=3&uuid=991db911-da8b-45c7-98b5-eb63e6162e8d&version=stable-2.7.1: dial tcp: lookup versioncheck.linkerd.io on 10.43.0.10:53: server misbehaving"

Any ideas?

[grampelberg]

What does describe look like on dev-covid-backend? The injector issues events when it tries to inject something.

[grampelberg]

Also, for questions like this it is easier to sort things out by jumping on slack.

[jeusdi]

Here logs of proxy-injector and linkerd-proxy:

$ kubectl logs -f --namespace linkerd linkerd-proxy-injector-7fcd58cb49-4hptz -c proxy-injector
time="2020-05-06T06:11:59Z" level=info msg="running version stable-2.7.1"
time="2020-05-06T06:12:00Z" level=info msg="waiting for caches to sync"
time="2020-05-06T06:12:01Z" level=info msg="caches synced"
time="2020-05-06T06:12:01Z" level=info msg="starting admin server on :9995"
time="2020-05-06T06:12:01Z" level=info msg="listening at :8443"

And linkerd-proxy logs:

$ kubectl logs -f --namespace linkerd linkerd-proxy-injector-7fcd58cb49-4hptz -c linkerd-proxy
time="2020-05-06T06:12:00Z" level=info msg="running version stable-2.7.1"
[     0.286506429s]  INFO linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.288720416s]  INFO linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.288799796s]  INFO linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.288835088s]  INFO linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.288872228s]  INFO linkerd2_proxy: Local identity is linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.288907372s]  INFO linkerd2_proxy: Identity verified via linkerd-identity.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.288939452s]  INFO linkerd2_proxy: Destinations resolved via linkerd-dst.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.312403177s]  INFO inbound: linkerd2_app_inbound: Serving listen.addr=0.0.0.0:4143
[     7.624543521s]  WARN inbound:accept{peer.addr=10.42.2.1:35226}:source{target.addr=10.42.2.167:9995}: linkerd2_app_core::errors: Failed to proxy request: request timed out
[    20.178084176s]  INFO daemon:identity: linkerd2_app: Certified identity: linkerd-proxy-injector.linkerd.serviceaccount.identity.linkerd.cluster.local

As you can see, it's getting a:

WARN inbound:accept{peer.addr=10.42.2.1:35226}:source{target.addr=10.42.2.167:9995}: linkerd2_app_core::errors: Failed to proxy request: request timed out

[olix0r]

This looks to me like the k8s api-server is sending requests to the injector pod before it passes its readiness check. The proxy won't pass a readiness check until the identity is acquired (at the 20s mark in this case).

@grampelberg Do we expect the api-server to honor readiness checks for admission controllers? Should we instead bind this to the liveness check?

[grampelberg]

AFAIK yes, as it is talking to the service which gets its endpoints added by passing the readiness check.

[grampelberg]

@jeusdi I still need:

What does describe look like on dev-covid-backend?

[jeusdi]

Sorry, here kubectl describe and also yaml:

$ kubectl describe deployment dev-covid-backend
Name:                   dev-covid-backend
Namespace:              covid
CreationTimestamp:      Wed, 06 May 2020 12:38:59 +0200
Labels:                 app.kubernetes.io/component=service
                        app.kubernetes.io/instance=backend-dev
                        app.kubernetes.io/managed-by=skaffold-v1.9.0
                        app.kubernetes.io/name=backend
                        app.kubernetes.io/part-of=covid
                        app.kubernetes.io/version=1.0.0
                        skaffold.dev/builder=local
                        skaffold.dev/cleanup=true
                        skaffold.dev/deployer=kustomize
                        skaffold.dev/docker-api-version=1.39
                        skaffold.dev/namespace=covid
                        skaffold.dev/profile.0=dev
                        skaffold.dev/run-id=75b08c6e-5e25-423d-8eb1-14a06dc00a72
                        skaffold.dev/tag-policy=git-commit
                        skaffold.dev/tail=true
                        stage=development
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app.kubernetes.io/component=service,app.kubernetes.io/instance=backend-dev,app.kubernetes.io/name=backend,app.kubernetes.io/part-of=covid,app.kubernetes.io/version=1.0.0,stage=development
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:       app.kubernetes.io/component=service
                app.kubernetes.io/instance=backend-dev
                app.kubernetes.io/managed-by=skaffold-v1.9.0
                app.kubernetes.io/name=backend
                app.kubernetes.io/part-of=covid
                app.kubernetes.io/version=1.0.0
                skaffold.dev/builder=local
                skaffold.dev/cleanup=true
                skaffold.dev/deployer=kustomize
                skaffold.dev/docker-api-version=1.39
                skaffold.dev/namespace=covid
                skaffold.dev/profile.0=dev
                skaffold.dev/run-id=75b08c6e-5e25-423d-8eb1-14a06dc00a72
                skaffold.dev/tag-policy=git-commit
                skaffold.dev/tail=true
                stage=development
  Annotations:  prometheus.io/path: /actuator/prometheus
                prometheus.io/port: 8080
                prometheus.io/scrape: true
  Containers:
   backend:
    Image:      registry.local:5000/covid-backend:e39b711-dirty@sha256:4d0251f76b5d7d918e914554acb93cb49140ba35aadeedf577d0d8125e09a20e
    Port:       8080/TCP
    Host Port:  0/TCP
    Environment Variables from:
      dev-covid-backend-config-b2mf6d22f2   ConfigMap  Optional: false
      dev-covid-backend-secrets-65dccgmc6g  Secret     Optional: false
    Environment:                            <none>
    Mounts:                                 <none>
  Volumes:                                  <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   dev-covid-backend-597cb6f8b6 (1/1 replicas created)
Events:          <none>

yaml:

$ kubectl get deployment -o=yaml dev-covid-backend
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"service","app.kubernetes.io/instance":"backend-dev","app.kubernetes.io/managed-by":"skaffold-v1.9.0","app.kubernetes.io/name":"backend","app.kubernetes.io/part-of":"covid","app.kubernetes.io/version":"1.0.0","skaffold.dev/builder":"local","skaffold.dev/cleanup":"true","skaffold.dev/deployer":"kustomize","skaffold.dev/docker-api-version":"1.39","skaffold.dev/namespace":"covid","skaffold.dev/profile.0":"dev","skaffold.dev/run-id":"75b08c6e-5e25-423d-8eb1-14a06dc00a72","skaffold.dev/tag-policy":"git-commit","skaffold.dev/tail":"true","stage":"development"},"name":"dev-covid-backend","namespace":"covid"},"spec":{"replicas":1,"selector":{"matchLabels":{"app.kubernetes.io/component":"service","app.kubernetes.io/instance":"backend-dev","app.kubernetes.io/name":"backend","app.kubernetes.io/part-of":"covid","app.kubernetes.io/version":"1.0.0","stage":"development"}},"template":{"metadata":{"annotations":{"prometheus.io/path":"/actuator/prometheus","prometheus.io/port":"8080","prometheus.io/scrape":"true"},"labels":{"app.kubernetes.io/component":"service","app.kubernetes.io/instance":"backend-dev","app.kubernetes.io/managed-by":"skaffold-v1.9.0","app.kubernetes.io/name":"backend","app.kubernetes.io/part-of":"covid","app.kubernetes.io/version":"1.0.0","skaffold.dev/builder":"local","skaffold.dev/cleanup":"true","skaffold.dev/deployer":"kustomize","skaffold.dev/docker-api-version":"1.39","skaffold.dev/namespace":"covid","skaffold.dev/profile.0":"dev","skaffold.dev/run-id":"75b08c6e-5e25-423d-8eb1-14a06dc00a72","skaffold.dev/tag-policy":"git-commit","skaffold.dev/tail":"true","stage":"development"}},"spec":{"containers":[{"envFrom":[{"configMapRef":{"name":"dev-covid-backend-config-b2mf6d22f2"}},{"secretRef":{"name":"dev-covid-backend-secrets-65dccgmc6g"}}],"image":"registry.local:5000/covid-backend:e39b711-dirty@sha256:4d0251f76b5d7d918e914554acb93cb49140ba35aadeedf577d0d8125e09a20e","name":"backend","ports":[{"containerPort":8080}]}]}}}}
  creationTimestamp: "2020-05-06T10:38:59Z"
  generation: 1
  labels:
    app.kubernetes.io/component: service
    app.kubernetes.io/instance: backend-dev
    app.kubernetes.io/managed-by: skaffold-v1.9.0
    app.kubernetes.io/name: backend
    app.kubernetes.io/part-of: covid
    app.kubernetes.io/version: 1.0.0
    skaffold.dev/builder: local
    skaffold.dev/cleanup: "true"
    skaffold.dev/deployer: kustomize
    skaffold.dev/docker-api-version: "1.39"
    skaffold.dev/namespace: covid
    skaffold.dev/profile.0: dev
    skaffold.dev/run-id: 75b08c6e-5e25-423d-8eb1-14a06dc00a72
    skaffold.dev/tag-policy: git-commit
    skaffold.dev/tail: "true"
    stage: development
  name: dev-covid-backend
  namespace: covid
  resourceVersion: "213380"
  selfLink: /apis/apps/v1/namespaces/covid/deployments/dev-covid-backend
  uid: 9d7958d0-de0b-4d54-8d36-3086da207912
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: service
      app.kubernetes.io/instance: backend-dev
      app.kubernetes.io/name: backend
      app.kubernetes.io/part-of: covid
      app.kubernetes.io/version: 1.0.0
      stage: development
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/path: /actuator/prometheus
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: service
        app.kubernetes.io/instance: backend-dev
        app.kubernetes.io/managed-by: skaffold-v1.9.0
        app.kubernetes.io/name: backend
        app.kubernetes.io/part-of: covid
        app.kubernetes.io/version: 1.0.0
        skaffold.dev/builder: local
        skaffold.dev/cleanup: "true"
        skaffold.dev/deployer: kustomize
        skaffold.dev/docker-api-version: "1.39"
        skaffold.dev/namespace: covid
        skaffold.dev/profile.0: dev
        skaffold.dev/run-id: 75b08c6e-5e25-423d-8eb1-14a06dc00a72
        skaffold.dev/tag-policy: git-commit
        skaffold.dev/tail: "true"
        stage: development
    spec:
      containers:
      - envFrom:
        - configMapRef:
            name: dev-covid-backend-config-b2mf6d22f2
        - secretRef:
            name: dev-covid-backend-secrets-65dccgmc6g
        image: registry.local:5000/covid-backend:e39b711-dirty@sha256:4d0251f76b5d7d918e914554acb93cb49140ba35aadeedf577d0d8125e09a20e
        imagePullPolicy: IfNotPresent
        name: backend
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-05-06T10:39:10Z"
    lastUpdateTime: "2020-05-06T10:39:10Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-05-06T10:38:59Z"
    lastUpdateTime: "2020-05-06T10:39:10Z"
    message: ReplicaSet "dev-covid-backend-597cb6f8b6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

[grampelberg]

We issue an event when the proxy injector either skips or injects a workload. It sounds like you're having issues with the api-server contacting the proxy injector. Are you running with a GKE private cluster or with firewall rules between your api server and kubelets?

I'd check in the apiserver logs, should be an explanation for what's going on in there.

[jeusdi]

I've created a local k3d cluster into my machine:

$ k3d create --name k3d --workers 2 --enable-registry \
    --env HTTP_PROXY=http://<proxy-ip>:<proxy-port> \
    --env HTTPS_PROXY=http://<proxy-ip>:<proxy-port> \
    --env NO_PROXY=.localhost,127.0.0.1,0.0.0.0,.local \
    --publish 80:80 --publish 443:443

When you say api-server do you mean kubernetes api server, or a linkerd service?

[grampelberg]

The kubernetes API server.

[grampelberg]

Note - moved over to discussions as this feels like a question right now.