401 unauthorized after some period of time

[medoedoff]

What went wrong?

What happened:

• We moved oncall to kubernetes cluster and we faced with issue, after some period of time around 2 days, integration between grafana and oncall breaks. I checked oncall logs, and could not found any issues, but found in celery logs:

2023-11-01 09:08:33,060 source=engine:celery worker=ForkPoolWorker-6 task_id=b29a733e-a514-45c1-bcce-06b44d1c3ad7 task_name=apps.grafana_plugin.tasks.sync.plugin_sync_organization_async name=apps.grafana_plugin.helpers.client level=WARNING Error connecting to api instance 401 Client Error: Unauthorized for url: https://grafana.company.com/api/access-control/users/permissions/search?actionPrefix=grafana-oncall-app

2023-11-01 09:08:33,060 source=engine:celery worker=ForkPoolWorker-6 task_id=b29a733e-a514-45c1-bcce-06b44d1c3ad7 task_name=apps.grafana_plugin.tasks.sync.plugin_sync_organization_async name=root level=INFO outbound latency=0.11716366000473499 status=401 method=HEAD url=https://grafana.company.com/api/access-control/users/permissions/search?actionPrefix=grafana-oncall-app slow=0

2023-11-01 09:08:33,175 source=engine:celery worker=ForkPoolWorker-6 task_id=b29a733e-a514-45c1-bcce-06b44d1c3ad7 task_name=apps.grafana_plugin.tasks.sync.plugin_sync_organization_async name=apps.grafana_plugin.helpers.client level=WARNING Error connecting to api instance 401 Client Error: Unauthorized for url: https://grafana.company.com/api/org

Could you please help us to fix this issue, after restarting a engine pod, integration between grafana and call begins to work, but again after some period of time it breaks, with the same error.

Oncall chart version: 1.3.12
Grafana version: 9.3.6

How do we reproduce it?

Chart values:

        base_url: redacted
        base_url_protocol: http
        imagePullSecrets:
          - name: docker-registry-key
        image:
          repository: gitlab.redacted.com:4567/redacted/docker-images/oncall
          tag: "v1.3.12"
          pullPolicy: IfNotPresent
        service:
          enabled: true
          type: ClusterIP
          port: 8080
        ingress:
          enabled: true
          annotations:
            kubernetes.io/ingress.class: "internal-nginx"
            nginx.ingress.kubernetes.io/ssl-redirect: "false"
            nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
            nginx.org/hsts: "false"
            nginx.org/hsts-include-subdomains: "false"
        ingress-nginx:
          enabled: false
        cert-manager:
          enabled: false
        mariadb:
          enbaled: true
          auth:
            existingSecret: mariadb
            database: redacted
            username: redacted
          primary:
            persistence:
              enabled: true
              storageClass: local-path
              size: 50Gi
        grafana:
          enabled: false
        externalGrafana:
          url: https://grafana.redacted.com
        rabbitmq:
          enabled: true
          auth:
            existingPasswordSecret: rabbitmq
        redis:
          enabled: true
          auth:
            existingSecret: redis
            existingSecretPasswordKey: passwordKey
        oncall:
          slack:
            enabled: true
            existingSecret: slack
            clientIdKey: clientIdKey
            clientSecretKey: clientSecretKey
            signingSecretKey: signingSecretKey
            redirectHost: https://oncall.redated.com
          twilio:
            accountSid: accountSid
            existingSecret: twilio
            authTokenKey: authTokenKey
            phoneNumberKey: phoneNumberKey
            verifySidKey: verifySidKey
            apiKeySidKey: apiKeySidKey
            apiKeySecretKey: apiKeySecretKey

Grafana OnCall Version

v1.3.12

Product Area

Auth

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

No response

[mderynck]

Likely thiss would be resolved by keeping mirageSecretKey constant through an external secret. As discovered #3607