cockroachdb / cockroachdb/cockroach-operator

Cluster fails to deploy when using Kyverno to mutate image name

Open
#1,101 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
318
Forks
104
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Hello,
I've deployed the operator in our cluster using the manifests, everything works as expected during install.

When deploying a cluster using the `example.yaml`, the `vcheck` job seems to run indefinitely. The job pod logs shows the version as expected. The operator logs don't point to any specific error at first, but after some time there seems to be some generic error messages.

[crdblog.log](https://github.com/user-attachments/files/19782005/crdblog.log)

After some trial and error I was able to determine that the issue seems to be from a Kyverno policy that we use to patch images to use our pullthrough cache.

```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: replace-dockerhub-registry-container
spec:
background: true
rules:
# Add the registry and tag if set implicitly
- name: set-default-registry-container
match:
resources:
kinds:
- Pod
mutate:
foreach:
# Containers
- list: "request.object.spec.containers"
patchStrategicMerge:
spec:
containers:
- (name): "{{ element.name }}"
(image): |-
!{{ images.containers."{{ element.name }}".referenceWithTag }}
image: |-
{{ images.containers."{{ element.name }}".referenceWithTag }}

# For all non official dockerhub images, replace the registry with the ECR pull through cache
- name: replace-dockerhub-registry-container
match:
resources:
kinds:
- Pod
preconditions:
any:
- key: "{{ request.object.spec.containers[*].image }}"
operator: AnyIn
value: "docker.io/*/*"
mutate:
foreach:
# Containers
- list: "request.object.spec.containers"
patchStrategicMerge:
spec:
containers:
- (name): "{{ element.name }}"
(image): |-
docker.io/*/*:{{images.containers."{{element.name}}".tag}}
image: our.pullthrough.cache.url/{{ images.containers."{{element.name}}".path }}:{{images.containers."{{element.name}}".tag}}

# The only remaining dockerhub images are offical and can be replaced with the ECR pull through cache
# Offical images require the prefix '/library'
- name: replace-official-dockerhub-registry-container
match:
resources:
kinds:
- Pod
preconditions:
any:
- key: "{{ request.object.spec.containers[*].image }}"
operator: AnyIn
value: "docker.io/*"
mutate:
foreach:
# Containers
- list: "request.object.spec.containers"
patchStrategicMerge:
spec:
containers:
- (name): "{{ element.name }}"
(image): |-
docker.io/*:{{images.containers."{{element.name}}".tag}}
image: our.pullthrough.cache.url/library/{{ images.containers."{{element.name}}".path }}:{{images.containers."{{element.name}}".tag}}
```
Removing this policy allowed the cluster to initiate properly.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the deployment from example.yaml with the supplied Kyverno ClusterPolicy, then compare the vcheck job logs with crdblog.log and the operator logs. Trace why the mutated image name leaves vcheck running indefinitely; done means the cluster initiates successfully while the image-mutation policy remains enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.