readiness probes of solr pod with `probesRequireAuth` set to `true` always return "Success" regardless of the actual result
- Dominant language
- Go
- Stars
- 283
- Forks
- 148
- PR merge metrics
- No merged PRs in 30d
Description
The Readiness Probes of the chart always return success as the command of the authenticated wget is always piped into a grep command which always has the return code `0` regardless weather they were able to access the Endpoint or not. This leads to the solr-pod starting clean and kept alive by k8s even though the authentication to the endpoints is actually failing. In my opinion this shall lead to the pod not becoming ready.
The same applies to the Liveness Probes respectively.
the generated yaml of the probes from the `solr pod` by the solr-operator:
```yaml
livenessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat
/etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory"
solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/system" 2>&1 | grep
-v JAVA_TOOL_OPTIONS
failureThreshold: 3
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 5
name: solrcloud-node
ports:
- containerPort: 8983
name: solr-client
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONS
```
```log
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONS
ERROR: Parse error :
Error 401 Bad credentials
HTTP ERROR 401 Bad credentials
URI:/solr/admin/info/health
STATUS:401
MESSAGE:Bad credentials
SERVLET:default
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0
```
```log
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1
Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
ERROR: Parse error :
Error 401 Bad credentials
HTTP ERROR 401 Bad credentials
URI:/solr/admin/info/health
STATUS:401
MESSAGE:Bad credentials
SERVLET:default
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
1
```
Downside is that without the `grep -v` the basicAuth Password may be logged. I can't figure out where it may be logged but, I assume the `grep -v` may have been introduce to suppress the logging of the credentials.
supplying the correct basic auth also renders the correct return code (only without the return code - suppressing grep):
```log
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1
Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
{
"responseHeader":{
"status":0,
"QTime":0},
"status":"OK"}
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Use the generated livenessProbe and readinessProbe commands as entry points; locate the chart or operator source that emits their sh -c pipelines and inspect how grep affects the exit status. Done means an authentication failure makes both probes fail while credential-suppression behavior remains safe, and valid credentials still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, shell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100