Automatically overwrite dashboard UIDs with a prescribed naming pattern, on the provider side

[sed-i]

Issue

• https://github.com/canonical/observability/blob/main/decision-records/2024-08-29--dashboard-uid-collision.md

Solution

Use shake_256 to generate a prescribed uid, overwriting any pre-existing uid.

In tandem with:

• Add dashboard uid generation helper cos-lib#114
• Autogenerate dashboard_alt_uid on the provider side grafana-agent-operator#224
• Replace GrafanaDashboard with LZMABase64 grafana-agent-k8s-operator#337

Context

• Afaict, the only user of the .add_dashboard method is https://github.com/canonical/cos-registration-server-k8s-operator/blob/6e156f0b1367f7cc6ce12d89f6a367bf491fea7f/src/charm.py#L240.
• Dahsboard templates (example) are rendered on the consumer side (grafana itself). The templates themselves are still valid json, so it's ok to json.loads them in the provider side.

Testing Instructions

First, prepare the charms locally:

1. Copy the grafana_dashboard lib from this PR into grafana-agent-operator from Autogenerate dashboard_alt_uid on the provider side grafana-agent-operator#224.
2. Copy cos-agent lib from Autogenerate dashboard_alt_uid on the provider side grafana-agent-operator#224 into zookeeper.
3. In zookeeper, update deps using the following commands. Then remove all the --hash from requirements.txt:

poetry update
poetry lock
poetry install
poetry export -f requirements.txt --output requirements.tx

4. Pack and deploy all charms as follows:

graph LR
subgraph lxd
zookeeper --- grafana-agent
zookeeper2 --- grafana-agent
end
subgraph microk8s
grafana-agent --- grafana
end

Loading

bundle: kubernetes
applications:
  graf:
    charm: ./grafana-k8s_ubuntu-20.04-amd64.charm
    scale: 1
    trust: true
--- # overlay.yaml
applications:
  graf:
    offers:
      graf:
        endpoints:
        - grafana-dashboard
        acl:
          admin: admin

default-base: [email protected]/stable
saas:
  graf:
    url: microk8s:admin/graf.graf
applications:
  ga2:
    charm: ./grafana-agent_ubuntu-22.04-amd64.charm
  zk:
    charm: ./zookeeper_ubuntu-22.04-amd64.charm
    num_units: 2
  zk2:
    charm: ./zookeeper_ubuntu-22.04-amd64.charm
    num_units: 2
relations:
- - ga2:cos-agent
  - zk:cos-agent
- - ga2:cos-agent
  - zk2:cos-agent
- - ga2:grafana-dashboards-provider
  - graf:grafana-dashboard

Upgrade Notes

• This change will break existing bookmarks to existing dashboards, because charms that fetch-lib this change would have the uid of all dashboards changed.
• If you have more than one app of the same charm deployed, they would all need to be refreshed together to avoid dashboard title collision. If you do not refresh all of them, grafana will output the following logs:
  • dashboard title is not unique in folder
  • dashboards provisioning provider has no database write permissions because of duplicates
  • Not saving new dashboard due to restricted database access

To enable automatic uid collision prevention for the dashboards in your charm:

1. If cos-lib (cosl) is pinned in your charm, bump it to 0.0.50 or higher.
2. Fetch-lib cos-agent. You should use 0.13 or higher.
3. Refresh your charm.
4. Refresh grafana-agent to revision 369+.

[sed-i]

I'd appreciate your review, @Guillaumebeuzeboc, @taurus-forever, @jdkandersson, @cbartz, @camille-rodriguez, @slapcat, @facundofc.

[jdkandersson]

I'll delegate to @cbartz who has the best understanding of the issue in our team, thank you for preparing the PR!

[sed-i]

This content isn't valid json, so had to modify them for the tests to pass.

[sed-i]

Should we bump LIBAPI?
On one hand, this change would break existing bookmarks to existing dashboards.
On the other hand, we would like charms to take this fix to UID collision via regular fetch-lib, without needing to manually fetch the next LIBAPI.

[sed-i]

Tested with:

graph LR
graf --- am
graf --- am2

Loading

Where alertmanager was first deployed from edge (with the "old" dashboard lib), then refreshed from path with the new lib from this PR.

bundle: kubernetes
applications:
  am:
    charm: local:alertmanager-k8s-1
    resources:
      alertmanager-image: 99
    scale: 2
    trust: true
  am2:
    charm: local:alertmanager-k8s-2
    resources:
      alertmanager-image: 99
    scale: 1
    trust: true
  graf:
    charm: grafana-k8s
    channel: latest/stable
    revision: 122
    resources:
      grafana-image: 70
      litestream-image: 45
    scale: 1
    trust: true
relations:
- - graf:grafana-dashboard
  - am:grafana-dashboard
- - graf:grafana-dashboard
  - am2:grafana-dashboard