Flaky tests TestHarnessRunIntegrationWithConfig/harness/crd-in-step

[porridge]

What happened:

https://github.com/kudobuilder/kuttl/actions/runs/8277018028/job/22646568404

=== NAME  TestHarnessRunIntegrationWithConfig/harness/crd-in-step
    logger.go:42: 07:33:41 | crd-in-step/0-crd | test step completed 0-crd
    logger.go:42: 07:33:41 | crd-in-step/1-create-mycrd | starting test step 1-create-mycrd
=== NAME  TestHarnessRunIntegrationWithConfig/harness/teststep-assert
    logger.go:42: 07:33:41 | teststep-assert/0-create | starting test step 0-create
=== NAME  TestHarnessRunIntegrationWithConfig/harness/crd-in-step
    case.go:366: failed in step 1-create-mycrd
    case.go:368: retrieving API resource for mycrd.example.com/v1beta1, Kind=MyCRD failed: the server could not find the requested resource
    logger.go:42: 07:33:41 | crd-in-step | crd-in-step events from ns kuttl-test-civil-barnacle:

What you expected to happen:

Since step 0-crd does wait for the CRD to be established, and kuttl does not seem to be using a cached discovery client, the error from API server when trying to create a CR is unexpected, especially in an unreplicated API server environment.

It would be useful to see API server audit log to know what exactly happened.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

This was in CI on this commit.

[porridge]

FWIW, this succeeded on a re-run.

[porridge]

Another:

=== CONT  TestHarnessRunIntegrationWithConfig/harness/crd-in-step
    logger.go:42: 08:32:39 | crd-in-step | Creating namespace: kuttl-test-primary-quail
    logger.go:42: 08:32:39 | crd-in-step/0-crd | starting test step 0-crd
    logger.go:42: 08:32:39 | crd-in-step/0-crd | CustomResourceDefinition:/mycrds.mycrd.example.com created
    logger.go:42: 08:32:39 | crd-in-step/0-crd | test step completed 0-crd
    logger.go:42: 08:32:39 | crd-in-step/1-create-mycrd | starting test step 1-create-mycrd
    case.go:399: failed in step 1-create-mycrd
    case.go:401: retrieving API resource for mycrd.example.com/v1beta1, Kind=MyCRD failed: the server could not find the requested resource
    logger.go:42: 08:32:39 | crd-in-step | crd-in-step events from ns kuttl-test-primary-quail:

[porridge]

Another:

=== CONT  TestHarnessRunIntegration/harness/crd-in-step
    logger.go:42: 08:49:08 | crd-in-step | Creating namespace: kuttl-test-smart-seasnail
=== NAME  TestHarnessRunIntegration/harness/create-or-update
    logger.go:42: 08:49:08 | create-or-update | Creating namespace: kuttl-test-measured-airedale
=== NAME  TestHarnessRunIntegration/harness/crd-in-step
    logger.go:42: 08:49:08 | crd-in-step/0-crd | starting test step 0-crd
=== NAME  TestHarnessRunIntegration/harness/delete-in-step
    logger.go:42: 08:49:08 | delete-in-step/0-create | test step completed 0-create
    logger.go:42: 08:49:08 | delete-in-step/1-create | starting test step 1-create
=== NAME  TestHarnessRunIntegration/harness/create-or-update
    logger.go:42: 08:49:08 | create-or-update/0-crd | starting test step 0-crd
=== NAME  TestHarnessRunIntegration/harness/crd-in-step
    logger.go:42: 08:49:08 | crd-in-step/0-crd | CustomResourceDefinition:/mycrds.mycrd.example.com created
=== NAME  TestHarnessRunIntegration/harness/create-or-update
    logger.go:42: 08:49:08 | create-or-update/0-crd | CustomResourceDefinition:/prowjobs.prow.example.com created
=== NAME  TestHarnessRunIntegration/harness/crd-in-step
    logger.go:42: 08:49:08 | crd-in-step/0-crd | test step completed 0-crd
    logger.go:42: 08:49:08 | crd-in-step/1-create-mycrd | starting test step 1-create-mycrd
    case.go:396: failed in step 1-create-mycrd
    case.go:398: retrieving API resource for mycrd.example.com/v1beta1, Kind=MyCRD failed: the server could not find the requested resource
=== NAME  TestHarnessRunIntegration/harness/create-or-update
    logger.go:42: 08:49:08 | create-or-update/0-crd | test step completed 0-crd
    logger.go:42: 08:49:08 | create-or-update/1-create-deployment | starting test step 1-create-deployment
=== NAME  TestHarnessRunIntegration/harness/crd-in-step
    logger.go:42: 08:49:08 | crd-in-step | crd-in-step events from ns kuttl-test-smart-seasnail: