Operator won't start services in single-stack IPv6 EKS
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 88
- Forks
- 66
- Avg merge
- 53m
- Merged PRs (30d)
- 1
Description
Version
pulp-operator is 1.0.0-beta.4. We do not have specific images for the pulp services that we are using, so whatever pulp-operator is pulling.
Describe the bug
The pulp-operator (and whatever containers it pulls to do its work) seem to assume that IPv4 will always be available, and will fail to run if only IPv6 is available. There does not seem to be any mechanism to override the IPv4 assumption.
To Reproduce
Steps to reproduce the behavior:
- Create an AWS EKS cluster with IPv6 only
- Deploy pulp-operator into a new namespace
- Apply a config:
apiVersion: repo-manager.pulpproject.org/v1beta2
kind: Pulp
metadata:
name: pulp
spec:
deployment_type: pulp
ingress_type: nodeport
pulp_settings:
api_root: "/pulp/"
allowed_export_paths:
- /tmp
allowed_import_paths:
- /tmp
telemetry: false
api:
replicas: 1
content:
replicas: 1
resource_requirements:
requests:
cpu: 150m
memory: 256Mi
limits:
cpu: 800m
memory: 1Gi
worker:
replicas: 1
resource_requirements:
requests:
cpu: 150m
memory: 256Mi
limits:
cpu: 800m
memory: 1Gi
web:
replicas: 1
resource_requirements:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 800m
memory: 1Gi
- Check the operator pod logs, you should see something like:
2024-12-14T15:04:07Z INFO repo_manager/database.go:129 Creating a new Database Service {"Service.Namespace": "pulp", "Service.Name": "pulp-database-svc"}
2024-12-14T15:04:07Z ERROR repo_manager/database.go:135 Failed to create new Database Service {"Service.Namespace": "pulp", "Service.Name": "pulp-database-svc", "error": "Service \"pulp-database-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
2024-12-14T15:04:07Z ERROR controller/controller.go:329 Reconciler error{"controller": "pulp", "controllerGroup": "repo-manager.pulpproject.org", "controllerKind": "Pulp", "Pulp": {"name":"pulp","namespace":"pulp"}, "namespace": "pulp", "name": "pulp", "reconcileID": "0fe0887b-37bf-41e8-9f99-45e1f2266549", "error": "Service \"pulp-database-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
Expected behavior
All Pulp pods should start and run, and Pulp should be functional.
Additional context
Some of our troubleshooting steps:
We attempted to use the kustomize mechanism to force IPv6 by setting the following in the manager_config_patch.yaml file, but there was no change:
spec:
template:
spec:
ipFamilyPolicy: SingleStack
ipFamilies:
- IPv6
There does not seem to be any available configuration within repo-manager.pulpproject.org/v1beta2 to influence the installed pods in any network-related configurations.
We found a few hardcoded instances of IPv4 within controllers/repo_manager/api.go, controllers/repo_manager/content.go, controllers/repo_manager/database.go, and controllers/telemetry.go, which seem like they should be a configuration option rather than a hardcoded value. We modified them and rebuilt, but make install deploy did not seem to use the new binary at all. Even if it did, the pod continued to generate the same errors.
Initially we were seeing the problem with the database service, so we decided to switch to an RDS instance, setting the appropriate secret config within our YAML file. The database service then seemed to be running, and the error had moved to the API service:
2024-12-18T17:27:09Z INFO repo_manager/utils.go:371 Creating a new pulp-api-svc Service {"Namespace": "pulp", "Name": "pulp-api-svc"}
2024-12-18T17:27:09Z ERROR repo_manager/utils.go:381 Failed to create new pulp-api-svc Service {"error": "Service \"pulp-api-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
2024-12-18T17:27:09Z ERROR controller/controller.go:329 Reconciler error{"controller": "pulp", "controllerGroup": "repo-manager.pulpproject.org", "controllerKind": "Pulp", "Pulp": {"name":"pulp","namespace":"pulp"}, "namespace": "pulp", "name": "pulp", "reconcileID": "e2fae6c3-4a30-4327-9327-f8885d755062", "error": "Service \"pulp-api-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
The pods were showing the following state:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pulp-api-6ff5f46cfd-xwlgj 0/1 CrashLoopBackOff 6 (4m35s ago) 11m
pulp-database-0 1/1 Running 0 5d18h
pulp-operator-controller-manager-669f6767cf-mtkts 2/2 Running 0 5d19h
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure on a single-stack IPv6 EKS cluster and start with the hardcoded IPv4 service definitions in controllers/repo_manager/api.go, controllers/repo_manager/content.go, controllers/repo_manager/database.go, and controllers/telemetry.go. Trace service creation through controllers/repo_manager/utils.go and verify that all generated Services use the cluster-supported address family and that the Pulp pods start and run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100