Use ADOT collector for metrics and potentially logging

[steveellis]

@jwfuller I think your first instinct was right about this. The cloudwatch agent container image referred to in the daemonset file is more than 2 years old. This agent code has been under active development.

https://github.com/aws-samples/amazon-cloudwatch-container-insights/blob/d60789889e0b679947e73d393618f6385a2a64ec/k8s-yaml-templates/cwagent-kubernetes-monitoring/cwagent-daemonset.yaml#L18

Think this should be latest.

I think we should indeed bring these yamls into our repo.

[steveellis]

Ref: https://hub.docker.com/r/amazon/cloudwatch-agent/tags

[steveellis]

Ok actually this is what we should be doing to solve this: https://github.com/aws-observability/aws-otel-helm-charts/tree/master/charts/adot-exporter-for-eks-on-ec2. This way we can deploy the agent with helm (and pulumi) and not have to use the shell script or the yamls at all.

This can also be used to deploy fluent bit.

What ADOT is and how it fits into the CloudWatch agent: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-otel.html.

[steveellis]

Ref #252 and #206.

[steveellis]

Ok configuring the chart it is possible to send data only to CW but it seems like AMP/Graphana might be easy to set up as well: https://github.com/aws-observability/aws-otel-helm-charts/blob/master/charts/adot-exporter-for-eks-on-ec2/documentation/metrics_to_cloudwatch_amp.md#sending-metrics-and-logs-to-cloudwatch-container-insights-only.

I think I'm going to try to get all of this going at once, metrics and logging using this helm chart, under this ticket.

[steveellis]

There's an issue with the aws helm chart. They've added some images to the chart docs and need to add a .helmignore file to ignore them. Since the chart is unusable and they say they are supporting it, I'd expect it to be fixed quickly: aws-observability/aws-otel-helm-charts#30.

[steveellis]

Metrics and logging are working on the scratch instance when deployed via script. The helm chart isn't working yet, the ADOT collector is however for both logging and metrics. Closes #174.

[steveellis]

Log retention should be part of this PR

cc @jwfuller

Log retention appears to be handled by this variable in the ConfigMap for FluentBit. If we deploy via kubectl apply (not the helm chart) this variable will need to be added since it isn't present by default.

That's all for retention of logs. Retention of metrics is another matter and it's not immediately clear where that would get controlled. It does seem like it should be a variable somewhere in the cwagentconfig.json here.

[steveellis]

What options are there for working with these aws yaml files in pulumi if we can't get the helm chart to work? Getting the helm chart working is being tracked here.

1. You can apply the yaml using a ConfigGroup like this: https://www.pulumi.com/docs/guides/adopting/from_kubernetes/
2. You could convert the yaml to pulumi like this: https://www.pulumi.com/blog/introducing-kube2pulumi/
3. There is the command package which lets you run a local command but I don't think it would reverse that operation on deletion: https://www.pulumi.com/registry/packages/command/

Only 2 would let you neatly handle passing parameters to the yaml (the cluster name for example is a runtime value).

If we apply the yaml in 1 it seems like we can change values inside that yaml using transformations.

[jwfuller]

It does seem like it should be a variable somewhere in the cwagentconfig.json

Maybe in these lines? The output conf section looks similar to in the ConfigMap for FluentBit