Avoid two grafana-agent related to one principal charm

[Abuelodelanada]

Enhancement Proposal

Although we know that this is not entirely in our hands, it depends on the limit established in the other charm, we need to find a way to avoid that more than one grafana-agent charm is related to a principal charm, because this can lead to multiple errors.

At least we need to set the second grafana-agent deployed into blocked status with a meaningful message.

For extra context read: https://discourse.charmhub.io/t/one-grafana-agent-charm-to-rule-them-all/16014

[lucabello]

We should think on whether we can bring this check into the charm.

One thing we could do is to check on the other side of the relation, and check ops.charm.meta of the other charm, and if we don't get limit: 1, then we send an error message. This would only work for cos-agent, not for juju-info.

Another option could be that when we render the grafana-agent config, we could add a comment with the path to charm.py of the charm that generated it. This way, grafana-agent could check whether the config was generated by a different grafana-agent, and send an error message/block.
Note that this idea is subject to race conditions; we'd need to tread carefully and possibly consider some os-level facilities to interact with mutex/locks.

We might want to do this directly in otel collector, and then think into how to apply it here as well.

[sed-i]

Related: juju/juju#18665
Duplicates #11