We are facing Error scraping metrics tls: failed to verify certificate: x509: certificate signed by unknown authority

[shbasha-clgx]

Hi Team,
We are trying to install otel collector using helm and trying to get the kubernetesAttributes:
enabled: true
kubeletMetrics:
enabled: true
hostMetrics:
enabled: true
logsCollection:
enabled: true
includeCollectorLogs:

configuration added in values.yaml how ever we are facing below error , attached file for your reference.

Kindly look into this and please assist

error scraperhelper/scrapercontroller.go:197 Error scraping metrics {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics", "error": "Get "
[https://xxxxxx:xxxx/stats/summary":]
tls: failed to verify certificate: x509: certificate signed by unknown authority", "scraper": "kubeletstats"}

we do not have self signed cert and need assistance on how to bypass this.

[shbasha-clgx]

@dmitryax Hi , can someone assist on this issue please

[shbasha-clgx]

@dmitryax Hi did anyone got a chance to go through the issue, we still see the same issue

[povilasv]

you can try enabling preset and also providing receiver config:

receivers:
  kubeletstats:
    insecure_skip_verify: true

Check this discussion around tls issues - #1034

[shbasha-clgx]

you can try enabling preset and also providing receiver config:

receivers:
  kubeletstats:
    insecure_skip_verify: true

Check this discussion around tls issues - #1034

@povilasv Thank you for your response
after adding
receivers:

kubeletstats:
insecure_skip_verify: true

we not seeing the tls error now but we are not able to get the cluster metrics in the otel collector logs although as exporter we put debug , below the values.yaml file we are using.

mode: daemonset

presets:
kubernetesAttributes:
enabled: true
kubeletMetrics:
enabled: true
hostMetrics:
enabled: true
logsCollection:
enabled: true
includeCollectorLogs: true

config:
exporters:
debug: {} # Enable OTLP HTTP exporter
service:
pipelines:
logs:
exporters:
- debug
receivers:
kubeletstats:
insecure_skip_verify: true

nodeSelector:
kubernetes.io/hostname: xxxxxx
tolerations:

• key: "node-role.kubernetes.io/control-plane"
  effect: "NoSchedule"

[shbasha-clgx]

@povilasv

Thank you for your response
after adding
receivers:

kubeletstats:
insecure_skip_verify: true

We are not seeing the tls error now but we are not able to get the cluster metrics in the otel collector logs although as exporter we put debug , below the values.yaml file we are using.

mode: daemonset

presets:
kubernetesAttributes:
enabled: true
kubeletMetrics:
enabled: true
hostMetrics:
enabled: true
logsCollection:
enabled: true
includeCollectorLogs: true

config:
exporters:
debug: {} # Enable OTLP HTTP exporter
service:
pipelines:
logs:
exporters:
- debug
receivers:
kubeletstats:
insecure_skip_verify: true

nodeSelector:
kubernetes.io/hostname: xxxxxx
tolerations:

• key: "node-role.kubernetes.io/control-plane"
  effect: "NoSchedule"

[shbasha-clgx]

@povilasv Hi , did you get a chance to look into this issue , can you please assist.

[povilasv]

Try enabling debug logs and seeing what is happening https://opentelemetry.io/docs/collector/internal-telemetry/#configure-internal-logs

[bixu]

below the values.yaml file we are using.

@shbasha-clgx, it'll help the community debug your issue if you paste formatted YAML.

You can do that by surrounding your YAML blocks with triple backticks, or selecting the YAML you've pasted and then clicking on the <> in the comment editor.

[D13410N3]

The same issue. How to deal with it?

[dmitryax]

@shbasha-clgx @D13410N3 Please post helm chart config (values.yaml), collector logs output, and k8s environment details (managed k8s? what version? etc.). Please use YAML blocks with triple backticks as mentioned by @bixu. I'd help to reproduce and potentially fix the problem

[cardoner1993]

Hi we are using the opentelemetry-operator version 0.78.0.
Using self signed certificate after some hours of activity we get the following error:
Failed to load live state: conversion webhook for opentelemetry.io/v1beta1, Kind=OpenTelemetryCollector failed: Post "https://otel-opentelemetry-operator-webhook.tools.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "opentelemetry-operator-operator-ca")

Our values yaml is as follows:

opentelemetry-operator:
  manager:
    collectorImage:
      repository: otel/opentelemetry-collector-k8s
  admissionWebhooks:
    certManager:
      enabled: false
    autoGenerateCert:
      enabled: true

Then:
ca-clusterissuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: root-secret

selfsigned-ca-certificate.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-ca
  namespace: tools
spec:
  isCA: true
  commonName: selfsigned-ca
  secretName: root-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io

selfsigned-clusterissuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

We started by allowing cert manager to create and sign the certificate as the documentation recommends but the same error was appearing immediately.

Thanks in advice.

[vstariradev]

Same issue here when trying helm chart version: 0.78.2

values.custom.yaml:

manager:
  collectorImage:
    repository: otel/opentelemetry-collector-k8s
  resources:
    limits:
      cpu: 300m
      memory: 512Gi
  serviceMonitor:
    enabled: true
admissionWebhooks:
  certManager:
    enabled: false