Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operator won't start services in single-stack IPv6 EKS #1380

Open
trinity-q opened this issue Dec 18, 2024 · 0 comments
Open

Operator won't start services in single-stack IPv6 EKS #1380

trinity-q opened this issue Dec 18, 2024 · 0 comments

Comments

@trinity-q
Copy link

Version
pulp-operator is 1.0.0-beta.4. We do not have specific images for the pulp services that we are using, so whatever pulp-operator is pulling.

Describe the bug
The pulp-operator (and whatever containers it pulls to do its work) seem to assume that IPv4 will always be available, and will fail to run if only IPv6 is available. There does not seem to be any mechanism to override the IPv4 assumption.

To Reproduce
Steps to reproduce the behavior:

  1. Create an AWS EKS cluster with IPv6 only
  2. Deploy pulp-operator into a new namespace
  3. Apply a config:
apiVersion: repo-manager.pulpproject.org/v1beta2
kind: Pulp
metadata:
  name: pulp
spec:
  deployment_type: pulp
  ingress_type: nodeport
  pulp_settings:
    api_root: "/pulp/"
    allowed_export_paths:
      - /tmp
    allowed_import_paths:
      - /tmp
    telemetry: false
  api:
    replicas: 1
  content:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 150m
        memory: 256Mi
      limits:
        cpu: 800m
        memory: 1Gi
  worker:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 150m
        memory: 256Mi
      limits:
        cpu: 800m
        memory: 1Gi
  web:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 800m
        memory: 1Gi
  1. Check the operator pod logs, you should see something like:
2024-12-14T15:04:07Z	INFO	repo_manager/database.go:129	Creating a new Database Service	{"Service.Namespace": "pulp", "Service.Name": "pulp-database-svc"}
2024-12-14T15:04:07Z	ERROR	repo_manager/database.go:135	Failed to create new Database Service	{"Service.Namespace": "pulp", "Service.Name": "pulp-database-svc", "error": "Service \"pulp-database-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
2024-12-14T15:04:07Z	ERROR	controller/controller.go:329	Reconciler error{"controller": "pulp", "controllerGroup": "repo-manager.pulpproject.org", "controllerKind": "Pulp", "Pulp": {"name":"pulp","namespace":"pulp"}, "namespace": "pulp", "name": "pulp", "reconcileID": "0fe0887b-37bf-41e8-9f99-45e1f2266549", "error": "Service \"pulp-database-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}

Expected behavior
All Pulp pods should start and run, and Pulp should be functional.

Additional context
Some of our troubleshooting steps:

We attempted to use the kustomize mechanism to force IPv6 by setting the following in the manager_config_patch.yaml file, but there was no change:

spec:
  template:
    spec:
      ipFamilyPolicy: SingleStack
      ipFamilies:
        - IPv6

There does not seem to be any available configuration within repo-manager.pulpproject.org/v1beta2 to influence the installed pods in any network-related configurations.

We found a few hardcoded instances of IPv4 within controllers/repo_manager/api.go, controllers/repo_manager/content.go, controllers/repo_manager/database.go, and controllers/telemetry.go, which seem like they should be a configuration option rather than a hardcoded value. We modified them and rebuilt, but make install deploy did not seem to use the new binary at all. Even if it did, the pod continued to generate the same errors.

Initially we were seeing the problem with the database service, so we decided to switch to an RDS instance, setting the appropriate secret config within our YAML file. The database service then seemed to be running, and the error had moved to the API service:

2024-12-18T17:27:09Z	INFO	repo_manager/utils.go:371	Creating a new pulp-api-svc Service	{"Namespace": "pulp", "Name": "pulp-api-svc"}
2024-12-18T17:27:09Z	ERROR	repo_manager/utils.go:381	Failed to create new pulp-api-svc Service	{"error": "Service \"pulp-api-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
2024-12-18T17:27:09Z	ERROR	controller/controller.go:329	Reconciler error{"controller": "pulp", "controllerGroup": "repo-manager.pulpproject.org", "controllerKind": "Pulp", "Pulp": {"name":"pulp","namespace":"pulp"}, "namespace": "pulp", "name": "pulp", "reconcileID": "e2fae6c3-4a30-4327-9327-f8885d755062", "error": "Service \"pulp-api-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}

The pods were showing the following state:

$ kubectl get pods
NAME                                                READY   STATUS             RESTARTS        AGE
pulp-api-6ff5f46cfd-xwlgj                           0/1     CrashLoopBackOff   6 (4m35s ago)   11m
pulp-database-0                                     1/1     Running            0               5d18h
pulp-operator-controller-manager-669f6767cf-mtkts   2/2     Running            0               5d19h
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant