Grafana dashboard should show commits

[jlewi]

The flux grafana dashboard shows the status of a deploy but not the commit.

I think it would be very useful to show the commit as well if possible.

[stefanprodan]

We don't export the source revision (branch/tag/sha/checksum) in our metrics, this would require changing the GitOps Toolkit runtime library and implement it in all controllers.

[kingdonb]

In the linkerd / buoyant cloud dashboard (I'm not sure which, I don't have a reference handy), they provide a view that is based on when deployments change, that does not depend on git commits at all. While that's not a grafana dashboard and linkerd surely does things that grafana won't be able to do natively, I wonder if we could do something similar?

There might be some enhancements that we can make, maybe if there are changes in the GitOps Toolkit to export more information in the metrics. I wonder if there have already been changes to support this? Should we investigate further? We did a cursory look over the metrics and we don't see any sign this support has been added, so I assume the answer is still the same.

I'm talking this over with @darkowlzz and we're thinking it would be nice to have a dashboard panel that tells about which git commits are in the cluster, by revision and by source. Something like the HelmRelease status dashboard that already exists. (Maybe those HelmRelease status values are exposed as metrics, and these values are not.)

I haven't explored the issue deeply, but I am starting to get into Grafana configuration and just beginning to go past what Flux's Prometheus monitoring guide explores. If there are some enhancements that we would like to see in there, perhaps it's something I can do?

[fsequeira1]

Hi, I am interested in working on this.

[kingdonb]

I see that fluxcd/notification-controller#322 was related to this issue but did not close it. (Should it have closed?)

I'm keen to try this out if it's out there and ready for users, or if it's close. 🎉

[fsequeira1]

@kingdonb I think we can close this one for now and if the changes on fluxcd/notification-controller#322 aren't enough @jlewi can open another issue and I will revisit it. What do you think?

[Legion2]

Can someone add an example how to configure grafana and alerts to see the commits in grafana. I created an alert but can't see anything in grafana.

[stefanprodan]

@Legion2 see:

• https://fluxcd.io/docs/guides/monitoring/#annotations
• https://fluxcd.io/docs/components/notification/provider/#grafana

[Legion2]

I created the Grafana Provider. But I think I also have to create an Alert, but this is not shown in the documentation.

[siegenthalerroger]

I do believe this can be closed now. However it'd be great to be able to change the tags that are given to the annotation.

[Legion2]

I finally got it working, the Grafana API Key needs the Admin role to work. But this is not documented.

[siegenthalerroger]

I actually have it working with the Editor role and it works fine.

[bizrad]

This wasn't obvious in the docs, but it appears that additional tags can be added to the annotations by using the Alert.spec.eventMetadata field (not the Provider object). Likely needed if you have multiple clusters reporting back to one Grafana instance. However, I also think the Provider could offer this as an option instead of it being on each alert.

https://github.com/fluxcd/notification-controller/blob/main/internal/notifier/grafana.go#L73