How partition key base work and how to use it?

[Flamentt]

Hi, I am readding this page and have a few question about Partition-key-based multi-tenancy: https://milvus.io/docs/multi_tenancy.md

As I read in the article "Multi-tenancy strategies", it mentioned that when using the Partition-key-based strategy, Milvus automatically manages partitions and supports up to 10M+ partitions. However, when I tried it, only 16 partitions were created.

I also tried using Ask AI and read some other documentation pages, and I found that I could specify the number of partitions, with the default being 16, but the maximum seems to be only 1024 or 4096 instead of 10M+.

This is making me a bit confused because I thought that when using Partition-key-based, the number of partitions would increase gradually according to the value of the partition field.

In my case, my collection has a date field, and I want to partition the data based on this field, with each day being a separate partition. However, when I created the collection with the following schema:

schema.add_field(
    field_name="date",
    datatype=DataType.VARCHAR,
    max_length=20,
    is_partition_key=True,
)

I then tried inserting 10k test data points (10k different values), and upon checking, only 16 partitions were created instead of the expected 10k partitions.

[yhmo]

The major purpose of "partition" is: split the data into different physical storage paths and control the search scope hence improving search performance significantly.

Let's say we have 1000 students, their exam scores are from 0 to 100. Now we want to find out a student whose score is 70.
If we put 1000 students into one partition, we need to scan all the 1000 scores.
If we split the scores into two partitions, scores from 0 to 49 are put into partition_1, and scores from 50 to 100 are put into partition_2. Then we need to scan about half of the scores.
If we split the scores into 10 partitions, then we only need to scan about 10% scores.

Will adding more scores change the number of partitions? No, doesn't change. If we split the 1000 scores into 10 partitions, then we add 500 scores, all of them are still in range [0, 100].

=============================================================================
Now, let's talk about partition key.
The usage:

schema.add_field(
    field_name="my_varchar", 
    datatype=DataType.VARCHAR, 
    max_length=512,
    is_partition_key=True,
)

client.create_collection(
    collection_name="my_collection",
    schema=schema,
    num_partitions=20
)

num_partitions defines how many partitions are created. All data will be split into these partitions.
Milvus internally generates a hash key for each record, the hash key is based on the value of the partition key, which means the same value of partition key will get the same hash key value.
Then, milvus splits the data into different partitions according to the value of math.mod(hash_key, num_partitions). No matter how many data you insert, the value of math.mod(hash_key, num_partitions) must be in the range [0, num_partitions).
Now, if you search with filter expression such as "[partition_key] == xxx", milvus also generates a hash key by the "xxx" and quickly know the target partition by math.mod(hash_key, num_partitions). It only searches in the target partition so that the search performance is much faster than full-scan.

You can change the num_partitions value to set partition number. The default value is 16. No need to set it to be a large value since too many partitions might bring overhead to milvus service. In my opinion, a value between 16 to 64 is recommended.

[xiaofan-luan]

Currently, each collection supports a maximum of 1,024 partitions. For multi-tenancy scenarios, you can create multiple collections, each with multiple partitions. However, creating a collection or partition consumes significant resources, meaning it's not feasible to create an unlimited number of physical partitions. For current release, we recommend a cluster has no more than 100K collection * partitions. We did some optimization in master branch so it could be larger in the future.

Partition keys provide a balanced approach by first dividing data into a limited number of physical partitions (e.g., the 16 partitions you see) and then further segmenting the data logically within each partition using filters. Since segments are logically isolated, a large number of tenants can be supported efficiently.

[Flamentt]

Thank you @xiaofan-luan and @yhmo for your really fast replies!

I have a better understanding of partitions now, but I still don't quite understand the term tenants. Is it an entity (row), or is it a value of a certain field that an entity can hold (e.g., in a "score" field, the values could be 1, 2, ..., 100, and these values 1–100 are the tenants)?

And in the table from the article "Multi-tenancy strategies", what does 10M+ tenants for "Partition-key-based" actually mean?

Thank you once again!

[yhmo]

In the doc, it mentioned "in many use cases, developers want to run one Milvus cluster and serve multiple tenants, such as a couple of product teams, or millions of end users", "The objective of multi-tenancy is to separate the data and resources from each other.".

We can separate the data in collection level, but milvus doesn't work well for a large amount of collections.

So, for 10M+ tenants, the recommended way is to handle it by partition-key. As I mentioned, with the hash+mod strategy, no matter how many entities are inserted, we don't need increase the number of partitions. 16 partitions work fine, 64 partitions also work fine, for 10M+ or even more tenants.

[Flamentt]

Thank you!

I have one more question about One partition per tenant part, it say that the number of tenants of the collection should not exceed the maximum number of partitions a collection can hold..

And a collection can have a maximum of 1,024 partitions. So we can have maximum of 1,024 tenants.

But in the table in above article, it show that Max. num. of tenants is 4,096 when using One partition per tenant.

[yhmo]

The max number of partitions is configured in the milvus.yaml:

rootCoord:
  maxPartitionNum: 1024

In old versions, the default value is 4096. We changed the default value to 1024 six months ago. So, 1024 is the correct number. I will update the doc later.

[Flamentt]

Thank you and @xiaofan-luan very much for your help! I think I have a pretty good understanding of partitioning now. I'll close the conversation!