Oncall 1.91: grafana-oncall-app can' connect to oncall

[bpedersen2]

What went wrong?

What happened:

• Upgraded oncall and oncall-app in an external grafana instance to v1.9.1
• The grafana app could not connect to the backend

error logs in the grafana pod:

2024-08-19 12:20:00.142	
logger=context userId=3 orgId=1 uname=<redacted> t=2024-08-19T10:20:00.142364973Z level=error msg="Request Completed" method=GET path=/api/plugins/grafana-oncall-app/resources/plugin/status status=500 remote_addr=10.42.0.198 time_ms=219 duration=219.648784ms size=144 referer=https://<redacted>/plugins/grafana-oncall-app handler=/api/plugins/:pluginId/resources/* status_source=downstream
2024-08-19 12:20:00.142	
logger=plugin.grafana-oncall-app t=2024-08-19T10:20:00.14224241Z level=error msg="Error validating oncall plugin settings" error="error setting up request headers: failed to parse JSON response: json: cannot unmarshal object into Go value of type []plugin.OnCallPermission "
2024-08-19 12:20:00.142	
logger=plugin.grafana-oncall-app t=2024-08-19T10:20:00.142227995Z level=error msg="Error getting user" error="failed to parse JSON response: json: cannot unmarshal object into Go value of type []plugin.OnCallPermission"
2024-08-19 12:20:00.141	
logger=context userId=11 orgId=1 uname=sa-1-autogen-1-oncall t=2024-08-19T10:20:00.141170208Z level=info msg="Request Completed" method=GET path=/api/access-control/users/3/permissions status=404 remote_addr=10.42.0.198 time_ms=17 duration=17.536455ms size=24 referer= handler=notfound status_source=server
2024-08-19 12:20:00.077	
logger=plugin.grafana-oncall-app t=2024-08-19T10:20:00.077178626Z level=info msg=GetUser user="map[Email:<redacted> Login:<redacted> Name:<redacted> Role:Admin]"

no error logs in oncall pods

What did you expect to happen:

• successful backend connection

How do we reproduce it?

1. Upgrade oncall/ oncall app in a setup with external grafana
   • maybe relevant: feature toogle accessControlOnCall is set to true
2. Try to access oncall
3. See the oncall is not configured correctly page

Grafana OnCall Version

v1.9.1

Product Area

Other

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

No response

[bpedersen2]

grafana version used: Grafana v11.3.0-192108 (65a7d8f023) (docker image: grafana/grafana:main)

[mderynck]

• Are you running Grafana Enterprise with a valid license? accessControlOnCall should only be used in that case
• Also externalServiceAccounts feature toggle should be enabled

[bpedersen2]

Yes, that was the solution. it is a oss installation. Until 1.8.13 it was running without problems with these settings. Maybe a hint in the releasenotes would be nice.

[senpro-ingwersenk]

Ran into the same issue. Thanks for pointing out accessControlOnCall - that was the culprit on our end as well.

[Tchoupinax]

As precision, do not use accessControlOnCall feature flag if you are using Grafana OSS.

[bavarian-ng]

If someone deploys Grafana and Oncall in Kubernetes via Grafana Operator and comes here due to the issue mentioned above:

the solution on our side was also to set this option in our grafana.yaml

kind: Grafana
metadata:
  name: {{ .Values.metadata.app_name }}
  namespace: {{ .Values.metadata.namespace }}
  labels:
    dashboards: "grafana"
spec:
  [...]
  config:
    [...]
    feature_toggles:
      accessControlOnCall: "false"

[bpedersen2]

That seems to be the correct workaround, see also #5255 and #5100