pulp / pulp/pulp-operator

Operator won't start services in single-stack IPv6 EKS

Open
#1,380 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Issue
Dominant language
Go
Stars
88
Forks
66
Avg merge
53m
Merged PRs (30d)
1

Description

Version
pulp-operator is 1.0.0-beta.4. We do not have specific images for the pulp services that we are using, so whatever pulp-operator is pulling.

Describe the bug
The pulp-operator (and whatever containers it pulls to do its work) seem to assume that IPv4 will always be available, and will fail to run if only IPv6 is available. There does not seem to be any mechanism to override the IPv4 assumption.

To Reproduce
Steps to reproduce the behavior:

  1. Create an AWS EKS cluster with IPv6 only
  2. Deploy pulp-operator into a new namespace
  3. Apply a config:
apiVersion: repo-manager.pulpproject.org/v1beta2
kind: Pulp
metadata:
  name: pulp
spec:
  deployment_type: pulp
  ingress_type: nodeport
  pulp_settings:
    api_root: "/pulp/"
    allowed_export_paths:
      - /tmp
    allowed_import_paths:
      - /tmp
    telemetry: false
  api:
    replicas: 1
  content:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 150m
        memory: 256Mi
      limits:
        cpu: 800m
        memory: 1Gi
  worker:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 150m
        memory: 256Mi
      limits:
        cpu: 800m
        memory: 1Gi
  web:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 800m
        memory: 1Gi
  1. Check the operator pod logs, you should see something like:
2024-12-14T15:04:07Z	INFO	repo_manager/database.go:129	Creating a new Database Service	{"Service.Namespace": "pulp", "Service.Name": "pulp-database-svc"}
2024-12-14T15:04:07Z	ERROR	repo_manager/database.go:135	Failed to create new Database Service	{"Service.Namespace": "pulp", "Service.Name": "pulp-database-svc", "error": "Service \"pulp-database-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
2024-12-14T15:04:07Z	ERROR	controller/controller.go:329	Reconciler error{"controller": "pulp", "controllerGroup": "repo-manager.pulpproject.org", "controllerKind": "Pulp", "Pulp": {"name":"pulp","namespace":"pulp"}, "namespace": "pulp", "name": "pulp", "reconcileID": "0fe0887b-37bf-41e8-9f99-45e1f2266549", "error": "Service \"pulp-database-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}

Expected behavior
All Pulp pods should start and run, and Pulp should be functional.

Additional context
Some of our troubleshooting steps:

We attempted to use the kustomize mechanism to force IPv6 by setting the following in the manager_config_patch.yaml file, but there was no change:

spec:
  template:
    spec:
      ipFamilyPolicy: SingleStack
      ipFamilies:
        - IPv6

There does not seem to be any available configuration within repo-manager.pulpproject.org/v1beta2 to influence the installed pods in any network-related configurations.

We found a few hardcoded instances of IPv4 within controllers/repo_manager/api.go, controllers/repo_manager/content.go, controllers/repo_manager/database.go, and controllers/telemetry.go, which seem like they should be a configuration option rather than a hardcoded value. We modified them and rebuilt, but make install deploy did not seem to use the new binary at all. Even if it did, the pod continued to generate the same errors.

Initially we were seeing the problem with the database service, so we decided to switch to an RDS instance, setting the appropriate secret config within our YAML file. The database service then seemed to be running, and the error had moved to the API service:

2024-12-18T17:27:09Z	INFO	repo_manager/utils.go:371	Creating a new pulp-api-svc Service	{"Namespace": "pulp", "Name": "pulp-api-svc"}
2024-12-18T17:27:09Z	ERROR	repo_manager/utils.go:381	Failed to create new pulp-api-svc Service	{"error": "Service \"pulp-api-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}
2024-12-18T17:27:09Z	ERROR	controller/controller.go:329	Reconciler error{"controller": "pulp", "controllerGroup": "repo-manager.pulpproject.org", "controllerKind": "Pulp", "Pulp": {"name":"pulp","namespace":"pulp"}, "namespace": "pulp", "name": "pulp", "reconcileID": "e2fae6c3-4a30-4327-9327-f8885d755062", "error": "Service \"pulp-api-svc\" is invalid: spec.ipFamilies[0]: Invalid value: \"IPv4\": not configured on this cluster"}

The pods were showing the following state:

$ kubectl get pods
NAME                                                READY   STATUS             RESTARTS        AGE
pulp-api-6ff5f46cfd-xwlgj                           0/1     CrashLoopBackOff   6 (4m35s ago)   11m
pulp-database-0                                     1/1     Running            0               5d18h
pulp-operator-controller-manager-669f6767cf-mtkts   2/2     Running            0               5d19h

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure on a single-stack IPv6 EKS cluster and start with the hardcoded IPv4 service definitions in controllers/repo_manager/api.go, controllers/repo_manager/content.go, controllers/repo_manager/database.go, and controllers/telemetry.go. Trace service creation through controllers/repo_manager/utils.go and verify that all generated Services use the cluster-supported address family and that the Pulp pods start and run.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.