On cluster expansion (dynamic nodes addition)

[andriytk]

I think the cluster expansion can be done iteratively in two stages:

1. Offline expansion.
2. Online expansion.

In the offline expansion, the cluster may be restarted after some new node is added. There are few things for this to be done to achieve this:

1. Update cfgen so that it does not regenerate the old FIDs of the cluster configuration entities on the expansion so that we can re-bootstrap the cluster without doing --mkfs.
2. Add some API for the expansion (like hctl node add or something).

Online expansion assumes that the cluster is not restarted during the addition of the node. For this, we might need to explore and utilise the Motr's API for reloading / updating the configuration (confd.xc) online without restarting the Motr processes. We might need support from Motr team here.

[johnbent]

Interesting thread @andriytk. The market expectation for object store is very high availability so growth needs to be done online.

[andriytk]

Yes, of course - the plan is to do it online when the feature is complete. The thread here is mainly to discuss - how are we going to implement it...

[mssawant]

@andriytk , Few tasks that I imagine which fall into the online cluster expansion are,

• Support partial CDF generation (I don't see much effort here other than provisioner)
• Support reading of the partial CDF via hctl interface (e.g. hctl cluster expand )
• All the newly added resources must be added to Consul KV,
  • cfgen needs to generate a partial consul-server-conf.json or update the existing consul-server-conf.json
  • cfgen needs to generate a partial consul-kv.json or update the existing consul-kv.json.
  • reload consul configuration.
• Start Consul, hax and motr services as per the new configuration
• Add the newly added node(s) to the running cluster
• Invoke motr configuration update

One tricky part here could be to decide whether to start consul process in server or client mode. But for now we can put this all in the partially generated CDF and decision can be made as we do now, by starting Consul server on the confd nodes and client on non-confd nodes.

[andriytk]

The partial CDF support is needed even for the offline expansion (as I mentioned in the beginning), isn't it?

My proposition is to do it iteratively in several stages in small steps which are showable (demonstratable) in the sprint demos and keep it under control instead of doing it all in a bulk which who knows when can be completed...

[rajanikantchirmade]

All the newly added resources must be added to Consul KV,
cfgen needs to generate a partial consul-server-conf.json or update the existing consul-server-conf.json
cfgen needs to generate a partial consul-kv.json or update the existing consul-kv.json.
reload consul configuration.

How about

• keep Motr configuration tree into consul-kv
• node-add updates consul-kv (running cluster)
• convert Motr-config from consul-kv. to confd.xc
• Copy updated confd.xc to all node where confd are runninfg
• Restart all confd in cluster.

[rajanikantchirmade]

I think confd looks redundant layer in presence of Consul-kv now.

1. Why can not we HaX cater requests from/to confc. And It keeps all Motr configuration with Consul-kv itself.
2. This may simplify Motr process by avoiding rconfc usage.
3. Consul-KV will keep Motr conf replicated and Motr conf always accessible through HaX

[andriytk]

I agree that having Consul we don't need rconfc on each client who wants to read the configuration. It's enough to just have confc on each client (some configuration cache is still needed - on each i/o op we do need to know where to send fops).

All the configuration is stored in a distributed quorum/consensus-based Consul's KV Store anyway, so confd services in Motr seem to be redundant now. @vvv was first who suggested it long ago.

It's just not quite clear how to update the configuration in this case. In rconfc case it is done with RM-read/write locking mechanism (all confcs have read locks which are revoked on the configuration update at confd side).

cc @nikitadanilov, @max-seagate.