How to add annotations to the pods in cluster

[MateoSegura]

Hello everyone, I am using vault side car injector feature to place the TLS certificates for my broker inside of each pod in a cluster deployment.

I've gotten this to work when I deploy a single instance of EMQX manually (meaning creating a k8s deployment by hand), but when I try to use the operator, I'm having trouble setting the annotations, here's what I have so far:

emqx-cluster.yaml

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx-cluster
spec:
  image: emqx/emqx:5.3.0
  imagePullPolicy: IfNotPresent
  
  serviceAccountName: my-service-account
  
  coreTemplate:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'my-role'
        vault.hashicorp.com/agent-inject-secret-server.key: 'kv/secret'
        vault.hashicorp.com/agent-inject-template-server.key: |
          {{ `{{- with secret "kv/secret" -}}
          {{ .Data.private_key }}
          {{- end }}` }}

But when I look at my pods, the annotations are not there, and no side car containers are injected. When I run kubectl describe deployment I get this:

Name:             emqx-cluster-core-84f45d97f5-0
Namespace:        default
Priority:         0
Service Account:  mqtt-broker
Node:             ip-10-0-23-48.us-west-2.compute.internal/10.0.23.48
Start Time:       Fri, 03 Jan 2025 23:45:41 +0000
Labels:           apps.emqx.io/db-role=core
                  apps.emqx.io/instance=emqx-cluster
                  apps.emqx.io/managed-by=emqx-operator
                  apps.emqx.io/pod-template-hash=84f45d97f5
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=emqx-cluster-core-84f45d97f5-7ddbb6cb99
                  role=core
                  statefulset.kubernetes.io/pod-name=emqx-cluster-core-84f45d97f5-0
Annotations:      <none> # Notice how the annotations did not get added to the pod
Status:           Running
IP:               10.0.7.114
IPs:
  IP:           10.0.7.114
Controlled By:  StatefulSet/emqx-cluster-core-84f45d97f5
Containers:
  emqx:
    Container ID:   containerd://dff53c5c3b3f64de7acf33d6847cfbde0916cf296514e3bc863bfd8bda94ac53
    Image:          emqx/emqx:5.3.0
    Image ID:       docker.io/emqx/emqx@sha256:18298f5a7ce054cfab28e1d90081885615c3e8a7a1fcd92b4af596e46b649a65
    Ports:          1883/TCP, 8883/TCP, 18083/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 03 Jan 2025 23:51:42 +0000
      Finished:     Fri, 03 Jan 2025 23:51:45 +0000
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Liveness:   http-get http://:dashboard/status delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:  http-get http://:dashboard/status delay=10s timeout=1s period=5s #success=1 #failure=12
    Environment:
      EMQX_DASHBOARD__LISTENERS__HTTP__BIND:  18083
      POD_NAME:                               emqx-cluster-core-84f45d97f5-0 (v1:metadata.name)
      EMQX_CLUSTER__DISCOVERY_STRATEGY:       dns
      EMQX_CLUSTER__DNS__RECORD_TYPE:         srv
      EMQX_CLUSTER__DNS__NAME:                emqx-cluster-headless.default.svc.cluster.local
      EMQX_HOST:                              $(POD_NAME).$(EMQX_CLUSTER__DNS__NAME)
      EMQX_NODE__DATA_DIR:                    data
      EMQX_NODE__ROLE:                        core
      EMQX_NODE__COOKIE:                      <set to the key 'node_cookie' in secret 'emqx-cluster-node-cookie'>  Optional: false
      EMQX_API_KEY__BOOTSTRAP_FILE:           "/opt/emqx/data/bootstrap_api_key"
    Mounts:
      /opt/emqx/data from emqx-cluster-core-data (rw)
      /opt/emqx/data/bootstrap_api_key from bootstrap-api-key (ro,path="bootstrap_api_key")
      /opt/emqx/etc/emqx.conf from bootstrap-config (ro,path="emqx.conf")
      /opt/emqx/log from emqx-cluster-core-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sl74s (ro)

[Rory-Z]

Hi, @MateoSegura What's the version of EMQX operator do you use ?

[MateoSegura]

Hello @Rory-Z, thanks for your response.

I am using emqx/emqx-operator-controller:2.2.26