如何让Query Node被快速替换？

[xiaobingxia-at]

Cluster 2.5.2, 600k个partition。因为各种原因要替换query node。比如version upgrade，更改config。
替换Query Node的过程是，启动一个新的query node，然后terminate一个现有的query node（该node会不断offload数据），那个新起来的query node会不断load新数据。

但是这个过程非常缓慢，大概有几小时。查看了query node的log，就是不停的在显示”migrate data..."
后来偶然的一次机会，发现被terminate的query node是每隔10分钟offload一批数据。被启动的query node也是每隔10分钟就load一批数据。然后就停止了。过十分钟再继续。

与此同时，我的checkBalanceInterval是600000，就是十分钟。
请问，migrate data的逻辑，是依靠balancer来完成的吗？ （先label segment，然后balancer每次要运行的时候创建task release / load segment)。
如果是这样，除了把checkBalanceInterval拉低（变频繁），还有其他办法吗？比如每次release / load的segment数目，能多一些吗？比如一次把所有segment release和load完，不要一次一次，持续几小时把一个query node关掉。

另外，我有意把checkBalanceInterval拉高，因为我不想让coordinator和query node过度负载。

[xiaofan-luan]

每次release/load的数据太多，一般会担心内存问题或者查询抖动。不过这个确实应该可以改进

@weiliu1031 please help on it

[xiaobingxia-at]

所以query node被替换的过程确实由balancer完成的吗？谢谢。

[weiliu1031]

Milvus uses a Balancer to migrate data from QueryNodes that are about to go offline. It provides the following parameters to control the frequency and batch size of balancing operations, ensuring minimal impact on query performance:

1. queryCoord.checkBalanceInterval (default: 3000ms) – Specifies that a balancing operation is triggered every 3000 milliseconds.
2. queryCoord.collectionBalanceSegmentBatchSize (default: 5) – Limits each balancing operation to transfer a maximum of 5 segments.
3. queryCoord.collectionBalanceChannelBatchSize (default: 1) – Limits each balancing operation to transfer a maximum of 1 channel.

By default, Milvus controls the workload of balancing operations primarily through batch size limits to reduce their impact on query performance. However, in your case, the queryCoord.checkBalanceInterval parameter has been further adjusted to limit the triggering frequency of balancing operations. This configuration may extend the overall balancing process.

Solutions:

1. Revert to Milvus's default rate-limiting strategy – Reduce the queryCoord.checkBalanceInterval to increase the frequency of balancing operations.
2. Continue with the current strategy – Increase the batch size for each balancing operation (adjust queryCoord.collectionBalanceSegmentBatchSize and queryCoord.collectionBalanceChannelBatchSize). However, this approach carries the risk of higher balancing workloads, which may impact query performance.

It is recommended to select the appropriate adjustment strategy based on your specific business requirements and sensitivity to query performance.
/cc @xiaofan-luan @xiaobingxia-at

[xiaobingxia-at]

@weiliu1031 thanks wei! A further question, since I have 600k partitions/segments, do you think iterating 600k segments per 3 seconds will make coordinator being overwhelmed? I see several issues now, for example, sometimes the indexing tasks assignments may pause for 10+ minutes for unknown reasons. So I intentionally increase the interval of every operations involving iterating all segments: like segment checking, balancer, indexing checking etc. Do you think it make sense to increase the interval of these operations to give coordinator an easy life?

Another concern, when I have balancer work in 3 second interval, I sometimes noticed that query node are super busy, I checked logs, there are so many segment loading/releasing. Do you think there is a risk of having balancer to work per 3 seconds?

[weiliu1031]

---

Adjusting the checker trigger interval slightly higher can be effective in your scenario, as it helps reduce the load on QueryCoord. However, there are some minor impacts to consider. For example, if the balance interval is set to 10 minutes, in the worst case, it may take up to 10 minutes for data to be migrated to a newly added QueryNode after scaling out.

Similarly, if the segment_checker or channel_checker intervals are adjusted to 10 minutes, loading a collection could experience delays—segments may not start loading until up to 10 minutes later. This might lead to issues such as load collection timeout.

There are ongoing optimizations to address load issues in multi-collection scenarios, and these improvements are expected to be released in future versions. In the meantime, if you encounter similar issues, you can temporarily mitigate them by slightly increasing the checker interval. Currently, 10 minutes is considered a sufficiently large value. However, if QueryCoord is not a bottleneck in your case, we recommend reducing the interval slightly—say, to 3 minutes—to minimize such delays.

Additionally, if a cluster is continuously performing load/release operations without new Insert/Delete data, it may indicate an abnormal condition that warrants attention. So far, we have not observed such issues in version 2.5.2. If you encounter similar problems, feel free to reach out to us via GitHub. We are more than happy to assist in investigating and troubleshooting any related issues.

Finally, to answer your question—triggering balance every 3 seconds is an expected configuration. The balance process will converge, and once the data distribution becomes even, there will no longer be any significant load/release overhead caused by balancing operations.

---

[xiaobingxia-at]

This is so helpful! thanks!