[Feature] Support for anti-affinity/affinity rules for the created machines

[sidharthsurana]

vsphere DRS supports defining anti-affinity / affinity rules for VMs. This feature is to add support for the user to specify affinity/anti-affinity grouping for the VMs.

Use case: User created 3 Machine object, and want all 3 VMs to run on different hosts to improve resiliency from host failures. This can easily be realized by creating the anti-affinity rule for the 3 VMs

[frapposelli]

/kind feature
/priority important-longterm
/assign @sidharthsurana

[sflxn]

This potentially can get done for v1alpha1, but it isn't a feature that's absolutely needs to go in for this release. If it gets done in time and the PR has been reviewed, we can make a judgement call at that time.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[moshloop]

Any reason why this can't be supported without DRS? Not sure DRS works across multiple clusters, and scheduling aross multiple clusters in a vcenter seems like a reasonable thing todo?

[akutz]

This may need to be punted to v1alpha2. If not we need to resource this work yesterday.

cc @frapposelli

[frapposelli]

@moshloop affinity/anti-affinity rules are defined within a DRS-enabled cluster, a cluster is also a fault-domain boundary for vSphere. Deploying across multiple clusters should be possible today using MachineSets.

@akutz this is potentially a noop (see kubernetes/cloud-provider-vsphere#179), but even if not, definitely needs to be punted to v1alpha2.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[moshloop]

/remove-lifecycle rotten

[sujeet-banerjee]

Any reason why this can't be supported without DRS? Not sure DRS works across multiple clusters, and scheduling across multiple clusters in a vcenter seems like a reasonable thing todo?

Short Answer: No. Without-DRS, it won't be effective.

Long Answer:
#1 One may use VM-Host rules without the "DRS enabled" on a vShpere cluster. However, the worker nodes will not be auto-migrated (or vMotioned) to support a balanced spread of the nodes across Hosts (i.e. HA in true sense). And CRDs do not expose Host details.
#2 That also means "vMotion" must be enabled for the Hosts.
More Details: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-7D3ABD21-4524-42E9-B7FE-6AAF6766433B.html

I have been working on a design-proposal for supporting Antiafinity/Affinity. Attached.
Spec_changes_for_AntiAffinity.docx
Test_n_Demo.pdf

[moshloop]

@sujeet-banerjee Can you create a google doc for commenting and review?

vMotion is something I believe should be turned off for kubernetes clusters as it conflicts with kubernetes view of the system. If a host fails then kubernetes should reschedule the pods that become not ready, cluster-api machineset should then detect unresponsive nodes, and create new ones.

Same applies to DRS, it should be turned off and kubernetes native components like descheduler used to move and rebalance workloads.

Without DRS or vMotion, machine affinity/antiAffinity can easily be implemented when creating a machine by listing all available hosts/clusters and running through rules list in the same way pod affinity is implemented.

[andrewsykim]

Can you create a google doc for commenting and review?

+1 to this, please use Google docs so we can comment/review :)

[sujeet-banerjee]

use Google Docs...

Could you point me to the shared location where I could add the doc?

@moshloop

Without DRS or vMotion, machine affinity/antiAffinity can easily be implemented when creating a machine by listing all available hosts/clusters and running through rules list in the same way pod affinity is implemented.

I somehow disagree on a few things,
#1 in my opinion, it's not a good idea to tie/specify host-details in the affinity-definition (machine/machine-set CRDs). As I understand, ESXi hosts may be added/taken-down at the will of the end-users, within a vSphere cluster.
#2 Similarly, Enabling/Disabling DRS should be end-users' will. An end-user may want to use DRS features, and CAPI/k8s should not be restrictive against using DRS features.

[sujeet-banerjee]

Proposal Doc: https://docs.google.com/document/d/1fNm53l5K0OfPrGc3zhjNDYzVQNBEc85uEJfNwtiDQrY

I would invite folks to review the proposal and add suggestions (if any).

Thanks,
Sujeet

[davidopp]

I believe
kubernetes/enhancements#997
kubernetes/enhancements#1127
are related?

[brysonshepherd]

@moshloop
Wouldn't it be better to vmotion, rather than build a whole new node on another vhost? To me that would lead longer time with pending pods.

[moshloop]

@brysonshepherd vMotion requires shared storage which is really expensive, slow and unreliable (when compared to just starting a new vm)

[moshloop]

, it's not a good idea to tie/specify host-details in the affinity-definition

antiAffinity using a topology key would work as new hosts, clusters and datancenters are added and removed:

e.g.

    vmAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution: 
            topologyKey: "vmware.io/hostname"

and to distribute nodes across multiple clusters

    vmAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution: 
            topologyKey: "vmware.io/clustername"

[brysonshepherd]

@moshloop

Some users may pay for that higher level of service for higher uptime.

In my organizations current use case, drs and vmotion are needed.

[moshloop]

higher uptime

Shared storage is much more likely to reduce uptime than it is to increase it, especially when your application is stateless and doesn't need persistent storage. Even when an application requires clustered or replicated storage, the control plane (kubernetes) should not share the same shared fault domain.

vMotion has it's advantages and is often the easiest solution from an operational overhead, but if you want to achieve the highest levels of resilience and uptime your kubernetes nodes shouldn't be using it.

[brysonshepherd]

@moshloop I'm not saying that vmotion is what I want, it is just faster than making a new node. If making a new node (along with rescheduling/starting up the pods) was faster, then that is what I would do. But it isn't. At least not that I'm aware of.

[moshloop]

The time to recover a pod doesn't need to be instant, just reasonable - You should be running multiple pods with enough headroom to survive the loss of one pod, If you lose a physical ESXi host than the VM doesn't get vMotion'ed, it gets booted on a different host, potentially goes through disk crash recovery and rejoins the cluster - in the meantime the pods running on the node may have already been detected as down and rescheduled.

[sujeet-banerjee]

The discussion moved to: https://docs.google.com/document/d/1fNm53l5K0OfPrGc3zhjNDYzVQNBEc85uEJfNwtiDQrY/edit#

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[akutz]

ping @pdaigle

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[moshloop]

/remove-lifecycle stale

[moshloop]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/lifecycle frozen

[jayunit100]

naively looking at this, and wondering,

Since @srm09 added https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/1182/files , would it be possibly to hijack the VerifyAffinityRule(ctx computeClusterContext, clusterName, hostGroupName, vmGroupName string) (Rule, error) { somehow, to do some kind of affinity/anti-affinity VM Spreading at a non-regional, but rather at just an ESXi'ish level ?

[srm09]

@jayunit100 I am not sure I quite understand what you are looking for here? Currently multi AZ allows you to attach VMs to specific hosts, if the host groups are setup to point to a particular host.
Can you elaborate on this ask?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[srm09]

/close
This has been implemented as the multi AZ feature using VSphereDeploymnetZone and VSphereFailureDomain CRDs.

[k8s-ci-robot]

@srm09: Closing this issue.

In response to this:

/close
This has been implemented as the multi AZ feature using VSphereDeploymnetZone and VSphereFailureDomain CRDs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sujeet-banerjee]

Does VSphereDeploymnetZone tie the machines (VMs) to specific ESXi hosts? As I understand, ESXi hosts may be added/taken down at the will of the end-users, within a vSphere cluster. Not sure if it's a good idea to tie/specify host details in the affinity definition (machine/machine-set CRDs).