High watermark memory usage never goes down (using a quorom queue)

[dsaintilma-flinks]

Describe the bug

High watermark memory usage never goes down (using a quorum queue) although there's very low to no message in the queue.

To Reproduce

• Setup and use a quorum queue

Config

Deployment (for the RabbitMQCluster)

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-cluster
  namespace: rabbitmq
spec:
  persistence:
    storage: 20Gi
    storageClassName: standard
  replicas: 3
  resources:
    limits:
      cpu: 500m
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 2Gi
  terminationGracePeriodSeconds: 60

Expected behavior
For the memory to purge itself but it keeps slowly going up.

Screenshots

Version and environment information

• RabbitMQ: 3.8.18
• RabbitMQ Cluster Operator: 1.8.1
• Kubernetes: 1.18.20-gke.501 (control plane), 1.18.20-gke.501 (nodes)
• Cloud provider or hardware configuration: GKE

[michaelklishin]

There is no evidence of a bug. No Memory breakdown was provided. This is a discussion, not a researched issue at this point.

Quorum queues keep up to 500 MiB of log segments in memory by default. You can reduce that to a drastically lower value.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

"Use a quorum queue" is a very brief description of what exactly was done but looking at the message rate chart, looks like the queue experiences very low publishing and consumption rates. In which case log truncation and WAL flushing performed by quorum queues will happen after a long period of time.

From the quorum queues doc guide:

Quorum queues use a write-ahead-log (WAL) for all operations. WAL operations are stored both in memory
and written to disk. 

When the current WAL file reaches a predefined limit, it is flushed to a WAL segment file on disk and the system
will begin to release the memory used by that batch of log entries. 
The segment files are then compacted over time as consumers acknowledge deliveries.
Compaction is the process that reclaims disk space.

The WAL file size limit at which it is flushed to disk can be controlled:

and

The value defaults to 512 MiB. This means that during steady load, the WAL table memory footprint can reach 512 MiB.

Steady load of 2-4 messages a second is, well, steady but it means that the above condition take a long time to materialize

Setting raft.wal_max_size_bytes to a value that is much lower, say, 16000000 (16 MB) would make WAL flushing and truncation happen much sooner, and limit peak memory usage by quorum queues. Such value is perfectly reasonable for quorum queues that don't experience ingress or egress message rates above 1K/second. In other words, this value can take you a long way.

[michaelklishin]

In addition, there are per-queue settings that can further help limit peak memory usage of quorum queues. A value of x-max-in-memory-bytes set to the max WAL segment file makes most sense to me.

[dsaintilma-flinks]

Hello Micheal,

Thank you for the very useful recommendations you have shared. I've setup the parameter raft.wal_max_size_bytes to 16MB and it does help. I'll explore the other parameters if necessary.

Cheers.