security.json not created in zookeeper
- Dominant language
- Go
- Stars
- 283
- Forks
- 148
- PR merge metrics
- No merged PRs in 30d
Description
Problem is similar to one reported in https://github.com/apache/solr-operator/issues/720
I tried install Solr Operator and Solr Cloud from scratch.
```yaml
# secrets
apiVersion: v1
data:
password:
username:
kind: Secret
metadata:
annotations:
argocd.argoproj.io/tracking-id: solr-9:/Secret:solr-9/solr-9-solrcloud-basic-auth
creationTimestamp: "2025-04-04T20:21:16Z"
name: solr-9-solrcloud-basic-auth
namespace: solr-9
type: kubernetes.io/basic-auth
---
apiVersion: v1
data:
security.json: kind: Secret
metadata:
annotations:
argocd.argoproj.io/tracking-id: solr-9:/Secret:solr-9/solr-9-solrcloud-security-secret-json
labels:
app.kubernetes.io/component: solr
app.kubernetes.io/instance: solr
app.kubernetes.io/name: solr
name: solr-9-solrcloud-security-secret-json
namespace: solr-9
type: Opaque
```
Security.json file
```json
{
"authentication": {
"class": "solr.MultiAuthPlugin",
"schemes": [{
"scheme": "basic",
"class":"solr.BasicAuthPlugin",
"blockUnknown": true,
"realm": "solr",
"forwardCredentials": false,
"credentials":{
"admin": ""
, "solr": ""
, "k8s-oper": ""
}
}]
},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{
"admin": ["admin", "k8s"],
"k8s-oper": ["k8s"],
"solr": ["users", "k8s"]
},
"permissions": [
{ "name": "k8s-probe-0", "role":"null", "collection": "null", "path":"/admin/info/system" },
{ "name": "k8s-probe-1", "role":"null", "collection": "null", "path":"/admin/info/health" },
{ "name": "k8s-status", "role":"k8s", "collection": "null", "path":"/admin/collections" },
{ "name": "k8s-metrics", "role":"k8s", "collection": "null", "path":"/admin/metrics" },
{ "name": "k8s-zk", "role":"k8s", "collection": "null", "path":"/admin/zookeeper/status" },
{ "name": "k8s-ping", "role":"k8s", "collection": "*", "path":"/admin/ping" },
{ "name": "k8s-replica-balancing", "role": "k8s", "collection": null, "path": "/____v2/cluster/replicas/balance" },
{ "name": "collection-admin-edit", "role": "k8s" },
{ "name": "read", "role":["admin","users"] },
{ "name": "update", "role":["admin"] },
{ "name": "core-admin-read", "role":["admin"] },
{ "name": "security-read", "role": ["admin"] },
{ "name": "security-edit", "role": ["admin"] },
{ "name": "all", "role":["admin"] }
]
}
}
```
As one can see Operator PODs come up first, Zookeeper PODs come up next and then Cloud PODs come up.
```shell
helm/solr-9 % kubectl get pods -w
NAME READY STATUS RESTARTS AGE
solr-operator-69f974cfc8-tzlcw 0/1 Running 0 2s
solr-operator-zookeeper-operator-79cb947c45-jkprq 1/1 Running 0 2s
solr-operator-69f974cfc8-tzlcw 1/1 Running 0 10s
.
.
helm/solr-9 % kubectl get pods -w
NAME READY STATUS RESTARTS AGE
solr-9-solrcloud-0 0/2 Pending 0 4s
solr-9-solrcloud-1 0/2 Pending 0 4s
solr-9-solrcloud-2 0/2 Pending 0 4s
solr-9-zookeeper-0 0/1 ContainerCreating 0 4s
solr-9-zookeeper-1 0/1 Pending 0 4s
solr-9-zookeeper-2 0/1 Pending 0 4s
solr-operator-69f974cfc8-tzlcw 1/1 Running 0 118s
solr-operator-zookeeper-operator-79cb947c45-jkprq 1/1 Running 0 118s
solr-9-solrcloud-1 0/2 Pending 0 5s
solr-9-solrcloud-1 0/2 Init:0/2 0 5s
solr-9-zookeeper-2 0/1 Pending 0 8s
solr-9-solrcloud-0 0/2 Pending 0 8s
solr-9-zookeeper-2 0/1 ContainerCreating 0 8s
solr-9-solrcloud-0 0/2 Init:0/2 0 8s
solr-9-solrcloud-2 0/2 Pending 0 9s
solr-9-solrcloud-2 0/2 Init:0/2 0 9s
solr-9-zookeeper-0 0/1 Running 0 12s
solr-9-solrcloud-1 0/2 Init:1/2 0 13s
solr-9-zookeeper-2 0/1 Running 0 13s
solr-9-solrcloud-0 0/2 Init:1/2 0 13s
solr-9-solrcloud-1 0/2 Init:1/2 0 14s
solr-9-solrcloud-0 0/2 Init:1/2 0 14s
solr-9-zookeeper-1 0/1 Pending 0 16s
solr-9-zookeeper-1 0/1 ContainerCreating 0 16s
solr-9-solrcloud-2 0/2 Init:1/2 0 17s
solr-9-solrcloud-2 0/2 Init:1/2 0 18s
solr-9-zookeeper-0 1/1 Running 0 22s
solr-9-zookeeper-2 1/1 Running 0 23s
solr-9-zookeeper-1 0/1 Running 0 24s
solr-9-zookeeper-1 1/1 Running 0 34s
solr-9-solrcloud-0 0/2 PodInitializing 0 45s
solr-9-solrcloud-2 0/2 PodInitializing 0 47s
solr-9-solrcloud-0 1/2 Running 0 48s
solr-9-solrcloud-1 0/2 PodInitializing 0 50s
solr-9-solrcloud-2 1/2 Running 0 51s
solr-9-solrcloud-1 1/2 Running 0 55s
solr-9-solrcloud-0 1/2 Running 0 58s
solr-9-solrcloud-0 2/2 Running 0 60s
solr-9-solrcloud-2 1/2 Running 0 63s
solr-9-solrcloud-1 1/2 Running 0 64s
solr-9-solrcloud-2 2/2 Running 0 64s
solr-9-solrcloud-1 2/2 Running 0 66s
.
.
.
##### Zookeeper has empty security.json ###############
I have no name!@solr-9-zookeeper-0:/$ zkCli.sh
/opt/bitnami/java/bin/java
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null zxid: -1
[zk: localhost:2181(CONNECTED) 0] get /solr/security.json
{}
```
Slor Cloud POD's **setup-zk** logs are as below
```
helm/solr-9 % kubectl logs solr-9-solrcloud-0 -c setup-zk
WARN - 2025-04-04 20:21:36.604; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 20:21:38.097; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ERROR: KeeperErrorCode = NoNode for /solr
WARN - 2025-04-04 20:21:43.898; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 20:21:45.301; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
Creating ZooKeeper path /solr on ZooKeeper at solr-9-zookeeper-0.solr-9-zookeeper-headless:2181,solr-9-zookeeper-1.solr-9-zookeeper-headless:2181,solr-9-zookeeper-2.solr-9-zookeeper-headless:2181
No security.json found. Put new security.json in ZK
```
I restarted Slor Cloud PODs (scale down the statefulset and let operator scale the PODs back.). Now setup-zk containers say the below:
```
helm/solr-9 % kubectl logs solr-9-solrcloud-0 -c setup-zk
WARN - 2025-04-04 22:14:10.453; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 22:14:12.626; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
configs
overseer
aliases.json
live_nodes
collections
overseer_elect
security.json
node_roles
No security.json found. Put new security.json in ZK
```
So, the only solution is manually add security.json because the operator never creates default one.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce a fresh Helm installation and inspect the Solr Cloud setup-zk container logs, especially the security.json handling shown in the issue. Verify the resulting ZooKeeper state with zkCli.sh and check that the expected security.json is created at /solr rather than requiring manual insertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes
- Domain
- devops, infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100