CNV-52722: Pass through extra VDDK configuration options to importer pod. #3572

mrnold · 2024-12-16T18:25:12Z

What this PR does / why we need it:
This pull request adds a new annotation "cdi.kubevirt.io/storage.pod.vddk.extraargs", referencing a ConfigMap that contains extra parameters to pass directly to the VDDK library. The use case is to allow tuning of asynchronous buffer counts for MTV as requested in CNV-52722. Testing has shown good results for cold migrations with:

VixDiskLib.nfcAio.Session.BufSizeIn64KB=16
VixDiskLib.nfcAio.Session.BufCount=4

These parameters are stored in a file whose path is passed to the VDDK via the nbdkit "config=" option. The file contents come from the referenced ConfigMap, and the ConfigMap is mounted to the importer pod as a volume.

Which issue(s) this PR fixes:
Fixes CNV-52722

Special notes for your reviewer:
As far as I can tell, a ConfigMap volume mount must be in the same namespace as the importer pod. So MTV will need to create or duplicate the ConfigMap to the same namespace as the DataVolume it creates.

Release note:

Allow extra VDDK configuration parameters to be passed to VDDK importer pods.

kubevirt-bot · 2024-12-16T18:25:23Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign awels for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mrnold · 2024-12-16T18:32:06Z

@mnecas Any concerns about the need to copy the ConfigMap to the importer namespace? I wasn't sure if that would make things awkward to use from the forklift side.

coveralls · 2024-12-16T18:41:56Z

coverage: 59.406% (+0.03%) from 59.373%
when pulling aacc61a on mrnold:cnv-52722
into 791bbd6 on kubevirt:main.

mnecas

Generally LGTM, I have added some a note and NP but nothing on my side

mnecas · 2024-12-16T19:48:47Z

pkg/image/nbdkit.go

+	withHidden, err := os.ReadDir(common.VddkArgsDir)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return "", nil


NP: Please add a comment that the user did not specify the vddk additional config

mnecas · 2024-12-16T19:50:17Z

pkg/image/nbdkit.go

@@ -228,6 +236,30 @@ func getVddkPluginPath() NbdkitPlugin {
 	return NbdkitVddkPlugin
 }

+// Extra VDDK configuration options are stored in a ConfigMap mounted to the
+// importer pod. Just look for the first file in the mounted directory, and


Just look for the first file just note we need to be sure to document this so user does not chain the configs to separate configmaps.

kubevirt-bot · 2024-12-16T19:51:58Z

@mnecas: changing LGTM is restricted to collaborators

In response to this:

Generally LGTM, I have added some a note and NP but nothing on my side

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mnecas · 2024-12-16T19:53:57Z

@mnecas Any concerns about the need to copy the ConfigMap to the importer namespace? I wasn't sure if that would make things awkward to use from the forklift side.

I think it should be okay, but also think we should document/verify.
Verify that the Forklift service account has enough permissions on the local cluster.
And check if the documentation specifies the permissions on a remote cluster.

mrnold · 2024-12-18T02:21:44Z

/retest

akalenyu

Looks good!

akalenyu · 2024-12-18T13:24:52Z

tests/datavolume_test.go

@@ -1254,6 +1276,30 @@ var _ = Describe("[vendor:[email protected]][level:component]DataVolume tests",
 					Message: "Import Complete",
 					Reason:  "Completed",
 				}}),
+			Entry("[test_id:5083]succeed importing VDDK data volume with extra arguments ConfigMap set", dataVolumeTestArguments{


Do you want to assert something in the final step of this test? like checking the config was applied?

For this test I wanted to make sure that the contents of the ConfigMap are present in the file, so the assertion happens in the vddk-test-plugin (the fgets/strcmp). Is there a better way to check the result from the importer? Like can this test read the pod logs or termination message?

oh okay I see

containerized-data-importer/tools/vddk-test/vddk-test-plugin.c

Line 50 in 625a9e9

if (strcmp(extras, "VixDiskLib.nfcAio.Session.BufSizeIn64KB=16") != 0) { // Must match datavolume_test

Yeah I think we can come up with a cleaner way,
You can either read pod logs or make this a part of the termination message struct

containerized-data-importer/pkg/importer/vddk-datasource_amd64.go

Lines 1011 to 1019 in 625a9e9

// GetTerminationMessage returns data to be serialized and used as the termination message of the importer.

func (vs *VDDKDataSource) GetTerminationMessage() *common.TerminationMessage {

return &common.TerminationMessage{

VddkInfo: &common.VddkInfo{

Version: vddkVersion,

Host: vddkHost,

},

}

}

I have an in-progress implementation to check the pod logs, but before I commit to that I am also going to try getting the result into the termination message to see if it's any cleaner. Either way it's definitely better to check the result from the test code than from the VDDK test plugin itself, that should get rid of a source of silent failures in CI.

The termination message approach should be cleaner, you would be asserting on a common.TerminationMessage struct that you got from the importer's API termination message

I tried this and putting the result in the termination message is only cleaner from this angle, the final assert. The rest of the implementation needed to have the importer scanning for test-only output from the nbdkit log just to save an arbitrary string value into the termination struct, which is fairly limited space that could probably be put to better use.

I think it ended up being nicer to just scan the log from the test, so the actual importer doesn't need to do extra stuff to accommodate the test.

Fair, thanks for trying 🙏

tools/vddk-test/vddk-test-plugin.c

pkg/image/nbdkit.go

tests/datavolume_test.go

akalenyu · 2024-12-18T13:31:00Z

doc/datavolumes.md

@@ -349,6 +349,13 @@ spec:
 [Get VDDK ConfigMap example](../manifests/example/vddk-configmap.yaml)
 [Ways to find thumbprint](https://libguestfs.org/nbdkit-vddk-plugin.1.html#THUMBPRINTS)

+#### Extra VDDK Configuration Options
+
+The VDDK library itself looks in a configuration file (such as `/etc/vmware/config`) for extra options to fine tune data transfers. To pass these options through to the VDDK, store the configuration file contents in a ConfigMap and add a `cdi.kubevirt.io/storage.pod.vddk.extraargs` annotation to the DataVolume specification. The ConfigMap will be mounted to the importer pod as a volume, and the first file in the mounted directory will be passed to the VDDK. This means that the ConfigMap must be placed in the same namespace as the DataVolume, and the ConfigMap should only have one file entry.


I guess you considered making this an API on the DataVolume, but, since you need a backport, you prefer the annotation?

Yes, it didn't seem worth changing the CRDs and all the generated stuff for just for this uncommon fine-tuning configuration option. I can certainly change the API if that would be better.

@mhenriks wdyt? since this is backporting I am leaning to the annotation as well, but I am not sure.. usually any annotation becomes an API that we forget about

Issue: The scale and perf team found a way how to improve the transfer speeds. Right now the only way to enable this feature is to set the v2v extra vars. The v2v extra vars pass the configuration to the virt-v2v and virt-v2v-in-place. The v2v extra vars configuration is general and not specific for VDDK. This causes the warm migration which uses the virt-v2v-in-place to fail as it does not use any VDDK parameters. Those parameters should be passed to the CNV CDI instead. Fix: Add a way to easily enable and configure the AIO. This feature is VDDK and provider-specific as it requires to have specific vSphere and VDDK versions. So we can't enable this feature globally nor by default. So this PR adds the configuration to the Provider spec settings and create a configmap with the necessary configuration and either mounts the configmap to the guest conversion pod for cold migration or passes the configmap name to the CDI DV annotation. Example: ``` apiVersion: forklift.konveyor.io/v1beta1 kind: Provider metadata: name: vsphere namespace: forklift spec: settings: sdkEndpoint: vcenter useVddkAioOptimization: 'true' vddkAioBufSize: 16 // optional defaults to 16 vddkAioBufCount: 4 // optional defaults to 4 vddkInitImage: 'quay.io/xiaodwan/vddk:8' type: vsphere ``` Ref: - https://issues.redhat.com/browse/MTV-1804 - kubevirt/containerized-data-importer#3572 - https://docs.redhat.com/en/documentation/migration_toolkit_for_virtualization/2.7/html-single/installing_and_using_the_migration_toolkit_for_virtualization/index#mtv-aio-buffer_mtv Signed-off-by: Martin Necas <[email protected]>

mrnold · 2024-12-18T21:17:43Z

/retest

Issue: The scale and perf team found a way how to improve the transfer speeds. Right now the only way to enable this feature is to set the v2v extra vars. The v2v extra vars pass the configuration to the virt-v2v and virt-v2v-in-place. The v2v extra vars configuration is general and not specific for VDDK. This causes the warm migration which uses the virt-v2v-in-place to fail as it does not use any VDDK parameters. Those parameters should be passed to the CNV CDI instead. Fix: Add a way to easily enable and configure the AIO. This feature is VDDK and provider-specific as it requires to have specific vSphere and VDDK versions. So we can't enable this feature globally nor by default. So this PR adds the configuration to the Provider spec settings and create a configmap with the necessary configuration and either mounts the configmap to the guest conversion pod for cold migration or passes the configmap name to the CDI DV annotation. Example: ``` apiVersion: forklift.konveyor.io/v1beta1 kind: Provider metadata: name: vsphere namespace: forklift spec: settings: sdkEndpoint: vcenter useVddkAioOptimization: 'true' vddkAioBufSize: 16 // optional defaults to 16 vddkAioBufCount: 4 // optional defaults to 4 vddkInitImage: 'quay.io/xiaodwan/vddk:8' type: vsphere ``` Ref: - https://issues.redhat.com/browse/MTV-1804 - kubevirt/containerized-data-importer#3572 - https://docs.redhat.com/en/documentation/migration_toolkit_for_virtualization/2.7/html-single/installing_and_using_the_migration_toolkit_for_virtualization/index#mtv-aio-buffer_mtv Signed-off-by: Martin Necas <[email protected]>

rwmjones · 2025-01-02T13:12:53Z

For interest only, here's an alternative to maintaining the fake VDDK plugin:

In nbdkit itself we are able to test the real nbdkit-vddk-plugin without VDDK, using a fake VDDK:

https://gitlab.com/nbdkit/nbdkit/-/blob/master/tests/dummy-vddk.c?ref_type=heads

which is compiled to libvixDiskLib.so.6:

https://gitlab.com/nbdkit/nbdkit/-/blob/9ed65418c57128d4bf372f39b9f98bf6ecbe470a/tests/Makefile.am

then you just have to point libdir= to the directory containing this file:

https://gitlab.com/nbdkit/nbdkit/-/blob/9ed65418c57128d4bf372f39b9f98bf6ecbe470a/tests/test-vddk.c#L61

mrnold · 2025-01-03T02:00:55Z

For interest only, here's an alternative to maintaining the fake VDDK plugin:

In nbdkit itself we are able to test the real nbdkit-vddk-plugin without VDDK, using a fake VDDK:

https://gitlab.com/nbdkit/nbdkit/-/blob/master/tests/dummy-vddk.c?ref_type=heads

which is compiled to libvixDiskLib.so.6:

https://gitlab.com/nbdkit/nbdkit/-/blob/9ed65418c57128d4bf372f39b9f98bf6ecbe470a/tests/Makefile.am

then you just have to point libdir= to the directory containing this file:

https://gitlab.com/nbdkit/nbdkit/-/blob/9ed65418c57128d4bf372f39b9f98bf6ecbe470a/tests/test-vddk.c#L61

This is cool, I will look into this. I think it would still require maintaining and updating an equivalent image to hold libvixDiskLib.so.6, but it would let me get rid of this test plugin check.

The VDDK library itself accepts infrequently-used arguments in a configuration file, and some of these arguments have been tested to show a significant transfer speedup in some environments. This adds an annotation that references a ConfigMap holding the contents of this VDDK configuration file, and mounts it to the importer pod. The first file in the mounted directory is passed to the VDDK. Signed-off-by: Matthew Arnold <[email protected]>

Signed-off-by: Matthew Arnold <[email protected]>

Instead of listing the mounted VDDK arguments directory and filtering out hidden files, just hard-code the expected file name and ConfigMap key. Signed-off-by: Matthew Arnold <[email protected]>

Put this in import_test and assert the values there, instead of in the VDDK test plugin. The VDDK plugin logs the given values, and then the test scans the log for what it expects to see. Signed-off-by: Matthew Arnold <[email protected]>

Signed-off-by: Matthew Arnold <[email protected]>

mrnold · 2025-01-08T02:50:28Z

/test pull-cdi-unit-test

mrnold · 2025-01-08T02:50:36Z

/test pull-cdi-goveralls

akalenyu · 2025-01-08T08:51:13Z

tests/import_test.go

@@ -189,6 +189,63 @@ var _ = Describe("[rfe_id:1115][crit:high][vendor:[email protected]][level:compo
 			Expect(importer.DeletionTimestamp).To(BeNil())
 		}
 	})
+
+	It("[test_id:6689]succeed importing VDDK data volume with extra arguments ConfigMap set", Label("VDDK"), func() {


Would you mind putting this under a separate describe so it doesn't inherit the "Serial" part?
(I think this test can run in parallel to others just fine)

You can also change [test_id:6689] to [test_id:XXXX]; I don't think it works well if this entry is missing from the ID database

Sure, done.

akalenyu · 2025-01-08T08:52:55Z

tests/framework/vddk.go

@@ -12,6 +12,38 @@ import (
 	"kubevirt.io/containerized-data-importer/tests/utils"
 )

+// CreateVddkDataVolume returns a VDDK data volume
+func (f *Framework) CreateVddkDataVolume(dataVolumeName, size, url string) *cdiv1.DataVolume {


How come something like this didn't already exist?

It does, but only in datavolume_test. I moved this test to import_test for some reason, I guess because I couldn't shoehorn the log scanning nicely into the big testDataVolume table. I moved it back to get rid of this change, it seems like it doesn't really belong in import_test any more than datavolume_test.

Signed-off-by: Matthew Arnold <[email protected]>

akalenyu · 2025-01-09T09:33:26Z

tests/datavolume_test.go

+			Expect(err).ToNot(HaveOccurred())
+			for _, option := range vddkConfigOptions {
+				By(fmt.Sprintf("Check for configuration value %s in nbdkit logs", option))
+				Expect(strings.Contains(logs, option)).To(BeTrue())


You can use the pod logs API instead of kubectl, check this out: #3184
And if you use .Should(ContainSubstring I am pretty sure it'll print the logs so we can properly understand why CI fails

Done, the end of the log gets truncated though. I am going to try showing the whole log temporarily and see if it offers any insights.

So, it passes locally?

Yes, it passes locally. It also correctly fails if I add another ContainSubstring line, and I can see the target strings in the log output.

Interesting, the config argument is not getting passed to nbdkit:

I0109 19:07:46.993477 1 nbdkit.go:311] Start nbdkit with: ['--foreground' '--readonly' '--exit-with-parent' '-U' '/tmp/nbd.sock' '--pidfile' '/tmp/nbd.pid' '--filter=retry' '--filter=cacheextents' '/opt/testing/libvddk-test-plugin.so' 'libdir=/opt/vmware-vix-disklib-distrib' 'server=vcenter.cdi-ukbxj:8989' 'user=user' 'password=+/tmp/password2397077821' 'thumbprint=testprint' 'vm=moref=vm-21' '--verbose' '-D' 'nbdkit.backend.datapath=0' '-D' 'vddk.datapath=0' '-D' 'vddk.stats=1' 'file=[teststore] testvm/testdisk.vmdk']

I must have a mistake further up somewhere.

I see, I didn't get the annotations copied over correctly through the import populator stuff. The importer prime doesn't get the annotations, so I guess the pod doesn't get the ConfigMap mounted. I can reproduce this locally now.

I see, yeah usually the hot path is CSI storage (and thus populators). I assume you were not setting any KUBEVIRT_STORAGE env var, and thus falling back to legacy population with the local storage class.

Signed-off-by: Matthew Arnold <[email protected]>

kubevirt-bot · 2025-01-09T23:25:23Z

@mrnold: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cdi-unit-test	`488849f`	link	true	`/test pull-cdi-unit-test`
pull-cdi-verify-go-mod	`488849f`	link	true	`/test pull-cdi-verify-go-mod`
pull-cdi-generate-verify	`488849f`	link	true	`/test pull-cdi-generate-verify`
pull-containerized-data-importer-e2e-istio	`488849f`	link	true	`/test pull-containerized-data-importer-e2e-istio`
pull-containerized-data-importer-e2e-hpp-latest	`488849f`	link	true	`/test pull-containerized-data-importer-e2e-hpp-latest`
pull-containerized-data-importer-e2e-hpp-previous	`488849f`	link	true	`/test pull-containerized-data-importer-e2e-hpp-previous`
pull-containerized-data-importer-e2e-nfs	`488849f`	link	true	`/test pull-containerized-data-importer-e2e-nfs`
pull-containerized-data-importer-e2e-upg	`488849f`	link	true	`/test pull-containerized-data-importer-e2e-upg`
pull-containerized-data-importer-e2e-ceph	`488849f`	link	true	`/test pull-containerized-data-importer-e2e-ceph`
pull-containerized-data-importer-e2e-ceph-wffc	`488849f`	link	true	`/test pull-containerized-data-importer-e2e-ceph-wffc`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Dec 16, 2024

kubevirt-bot added the size/L label Dec 16, 2024

kubevirt-bot requested review from awels and mhenriks December 16, 2024 18:25

mnecas approved these changes Dec 16, 2024

View reviewed changes

akalenyu reviewed Dec 18, 2024

View reviewed changes

mnecas mentioned this pull request Dec 18, 2024

MTV-1804 | Implement VDDK AIO buffer configuration kubev2v/forklift#1280

Open

mrnold added 7 commits January 7, 2025 17:01

Add functional test for VDDK args annotation.

c044ed6

Signed-off-by: Matthew Arnold <[email protected]>

Add unit test for extra VDDK arguments annotation.

ccca97f

Signed-off-by: Matthew Arnold <[email protected]>

Add documentation for extra VDDK arguments.

d9ed451

Signed-off-by: Matthew Arnold <[email protected]>

Simplify new functional test annotation creation.

fa0a87a

Signed-off-by: Matthew Arnold <[email protected]>

Look for specific file instead of first file.

0fe6322

Instead of listing the mounted VDDK arguments directory and filtering out hidden files, just hard-code the expected file name and ConfigMap key. Signed-off-by: Matthew Arnold <[email protected]>

Move extra VDDK arguments functional test.

49cd30f

Put this in import_test and assert the values there, instead of in the VDDK test plugin. The VDDK plugin logs the given values, and then the test scans the log for what it expects to see. Signed-off-by: Matthew Arnold <[email protected]>

mrnold force-pushed the cnv-52722 branch from 4ce1c2d to 49cd30f Compare January 7, 2025 22:01

Clean up lint error.

e66b180

Signed-off-by: Matthew Arnold <[email protected]>

akalenyu reviewed Jan 8, 2025

View reviewed changes

Move VDDK configuration test back, change test ID.

1120d5b

Signed-off-by: Matthew Arnold <[email protected]>

akalenyu reviewed Jan 9, 2025

View reviewed changes

mrnold added 2 commits January 9, 2025 10:19

Avoid using kubectl for scanning nbdkit logs.

1a97ee5

Signed-off-by: Matthew Arnold <[email protected]>

Temporary: show whole nbdkit log after failure.

488849f

Signed-off-by: Matthew Arnold <[email protected]>

	// GetTerminationMessage returns data to be serialized and used as the termination message of the importer.
	func (vs VDDKDataSource) GetTerminationMessage() common.TerminationMessage {
	return &common.TerminationMessage{
	VddkInfo: &common.VddkInfo{
	Version: vddkVersion,
	Host: vddkHost,
	},
	}
	}

CNV-52722: Pass through extra VDDK configuration options to importer pod. #3572

Are you sure you want to change the base?

CNV-52722: Pass through extra VDDK configuration options to importer pod. #3572

Conversation

mrnold commented Dec 16, 2024

kubevirt-bot commented Dec 16, 2024

mrnold commented Dec 16, 2024

coveralls commented Dec 16, 2024 • edited Loading

mnecas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kubevirt-bot commented Dec 16, 2024

mnecas commented Dec 16, 2024

mrnold commented Dec 18, 2024

akalenyu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrnold commented Dec 18, 2024

rwmjones commented Jan 2, 2025

mrnold commented Jan 3, 2025

mrnold commented Jan 8, 2025

mrnold commented Jan 8, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akalenyu Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akalenyu Jan 12, 2025 • edited Loading

Choose a reason for hiding this comment

kubevirt-bot commented Jan 9, 2025

coveralls commented Dec 16, 2024 •

edited

Loading

akalenyu Jan 9, 2025 •

edited

Loading

akalenyu Jan 12, 2025 •

edited

Loading