How to ensure that linkerd-proxy metrics are scraped from ephemeral pods?

[alexklibisz]

I really like having the pod and namespace labels on route_response_total{direction="outbound", dst="service.namespace.svc.cluster.local:port"}. This is very valuable for tracking down the source of slow and erroneous requests to a service.

However, there seem to be many cases where the direction="outbound" metrics are not reliably scraped from ephemeral pods (i.e. jobs). The proxy-wait-before-exit-seconds setting works well for long-lived pods (i.e. web services), which have their lifecycle managed by k8s. But it seems that ephemeral pods ignore this setting? It doesn't seem to matter whether the main (i.e. non-sidecar) container exits successfully or with an error code.

As I understand it, ephemeral pods are somewhat antithetical to prometheus' scrape/pull model. The push gateway is a possible solution, but has a number of quirks.

So is there a recommended best practice to ensure that linkerd-proxy containers on ephemeral pods are scraped? Alternatively, is there a way to get the source pod and namespace as labels on the route_response_total{direction="inbound"} metric?

[adleong]

I don't know of any "best practices" for this particular scenario. Is it possible to wrap your job application in a script that sleeps for a period (at least one prometheus scrape interval) after the main process completes? This way the pod should remain alive long enough to get scraped. Just an idea to try.

There is no way currently to get source labels for inbound metrics. See #4101

[alexklibisz]

Is it possible to wrap your job application in a script that sleeps for a period (at least one prometheus scrape interval) after the main process completes?

Yes, technically possible, but would rather not task everyone in the org with remembering to do this. For now I'll most likely stick with using the route_response_total{direction="inbound"} metrics by default for dashboards and alerts and use route_response_total{direction="inbound"} for ad-hoc debugging.

In any case, thanks for confirming and answering promptly!