apache / apache/solr-operator

readiness probes of solr pod with `probesRequireAuth` set to `true` always return "Success" regardless of the actual result

Open
#799 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
283
Forks
148
PR merge metrics
No merged PRs in 30d

Description

The Readiness Probes of the chart always return success as the command of the authenticated wget is always piped into a grep command which always has the return code `0` regardless weather they were able to access the Endpoint or not. This leads to the solr-pod starting clean and kept alive by k8s even though the authentication to the endpoints is actually failing. In my opinion this shall lead to the pod not becoming ready.

The same applies to the Liveness Probes respectively.

the generated yaml of the probes from the `solr pod` by the solr-operator:
```yaml
livenessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat
/etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory"
solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/system" 2>&1 | grep
-v JAVA_TOOL_OPTIONS
failureThreshold: 3
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 5
name: solrcloud-node
ports:
- containerPort: 8983
name: solr-client
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONS
```

```log
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONS

ERROR: Parse error :

Error 401 Bad credentials

HTTP ERROR 401 Bad credentials

URI:/solr/admin/info/health
STATUS:401
MESSAGE:Bad credentials
SERVLET:default

solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0
```

```log
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1

Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory

ERROR: Parse error :

Error 401 Bad credentials

HTTP ERROR 401 Bad credentials

URI:/solr/admin/info/health
STATUS:401
MESSAGE:Bad credentials
SERVLET:default

solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
1
```

Downside is that without the `grep -v` the basicAuth Password may be logged. I can't figure out where it may be logged but, I assume the `grep -v` may have been introduce to suppress the logging of the credentials.

supplying the correct basic auth also renders the correct return code (only without the return code - suppressing grep):
```log
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1

Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
{
"responseHeader":{
"status":0,
"QTime":0},
"status":"OK"}
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Use the generated livenessProbe and readinessProbe commands as entry points; locate the chart or operator source that emits their sh -c pipelines and inspect how grep affects the exit status. Done means an authentication failure makes both probes fail while credential-suppression behavior remains safe, and valid credentials still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.