apache / apache/solr-operator

security.json not created in zookeeper

Open
#780 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Go
Stars
283
Forks
148
PR merge metrics
No merged PRs in 30d

Description

Problem is similar to one reported in https://github.com/apache/solr-operator/issues/720

I tried install Solr Operator and Solr Cloud from scratch.

```yaml
# secrets
apiVersion: v1
data:
password:
username:
kind: Secret
metadata:
annotations:
argocd.argoproj.io/tracking-id: solr-9:/Secret:solr-9/solr-9-solrcloud-basic-auth
creationTimestamp: "2025-04-04T20:21:16Z"
name: solr-9-solrcloud-basic-auth
namespace: solr-9
type: kubernetes.io/basic-auth

---
apiVersion: v1
data:
security.json: kind: Secret
metadata:
annotations:
argocd.argoproj.io/tracking-id: solr-9:/Secret:solr-9/solr-9-solrcloud-security-secret-json
labels:
app.kubernetes.io/component: solr
app.kubernetes.io/instance: solr
app.kubernetes.io/name: solr
name: solr-9-solrcloud-security-secret-json
namespace: solr-9
type: Opaque

```

Security.json file
```json
{
"authentication": {
"class": "solr.MultiAuthPlugin",
"schemes": [{
"scheme": "basic",
"class":"solr.BasicAuthPlugin",
"blockUnknown": true,
"realm": "solr",
"forwardCredentials": false,
"credentials":{
"admin": ""
, "solr": ""
, "k8s-oper": ""
}
}]
},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{
"admin": ["admin", "k8s"],
"k8s-oper": ["k8s"],
"solr": ["users", "k8s"]
},
"permissions": [
{ "name": "k8s-probe-0", "role":"null", "collection": "null", "path":"/admin/info/system" },
{ "name": "k8s-probe-1", "role":"null", "collection": "null", "path":"/admin/info/health" },
{ "name": "k8s-status", "role":"k8s", "collection": "null", "path":"/admin/collections" },
{ "name": "k8s-metrics", "role":"k8s", "collection": "null", "path":"/admin/metrics" },
{ "name": "k8s-zk", "role":"k8s", "collection": "null", "path":"/admin/zookeeper/status" },
{ "name": "k8s-ping", "role":"k8s", "collection": "*", "path":"/admin/ping" },
{ "name": "k8s-replica-balancing", "role": "k8s", "collection": null, "path": "/____v2/cluster/replicas/balance" },
{ "name": "collection-admin-edit", "role": "k8s" },
{ "name": "read", "role":["admin","users"] },
{ "name": "update", "role":["admin"] },
{ "name": "core-admin-read", "role":["admin"] },
{ "name": "security-read", "role": ["admin"] },
{ "name": "security-edit", "role": ["admin"] },
{ "name": "all", "role":["admin"] }
]
}

}
```

As one can see Operator PODs come up first, Zookeeper PODs come up next and then Cloud PODs come up.
```shell
helm/solr-9 % kubectl get pods -w
NAME READY STATUS RESTARTS AGE
solr-operator-69f974cfc8-tzlcw 0/1 Running 0 2s
solr-operator-zookeeper-operator-79cb947c45-jkprq 1/1 Running 0 2s
solr-operator-69f974cfc8-tzlcw 1/1 Running 0 10s

.
.

helm/solr-9 % kubectl get pods -w
NAME READY STATUS RESTARTS AGE
solr-9-solrcloud-0 0/2 Pending 0 4s
solr-9-solrcloud-1 0/2 Pending 0 4s
solr-9-solrcloud-2 0/2 Pending 0 4s
solr-9-zookeeper-0 0/1 ContainerCreating 0 4s
solr-9-zookeeper-1 0/1 Pending 0 4s
solr-9-zookeeper-2 0/1 Pending 0 4s
solr-operator-69f974cfc8-tzlcw 1/1 Running 0 118s
solr-operator-zookeeper-operator-79cb947c45-jkprq 1/1 Running 0 118s
solr-9-solrcloud-1 0/2 Pending 0 5s
solr-9-solrcloud-1 0/2 Init:0/2 0 5s
solr-9-zookeeper-2 0/1 Pending 0 8s
solr-9-solrcloud-0 0/2 Pending 0 8s
solr-9-zookeeper-2 0/1 ContainerCreating 0 8s
solr-9-solrcloud-0 0/2 Init:0/2 0 8s
solr-9-solrcloud-2 0/2 Pending 0 9s
solr-9-solrcloud-2 0/2 Init:0/2 0 9s
solr-9-zookeeper-0 0/1 Running 0 12s
solr-9-solrcloud-1 0/2 Init:1/2 0 13s
solr-9-zookeeper-2 0/1 Running 0 13s
solr-9-solrcloud-0 0/2 Init:1/2 0 13s
solr-9-solrcloud-1 0/2 Init:1/2 0 14s
solr-9-solrcloud-0 0/2 Init:1/2 0 14s
solr-9-zookeeper-1 0/1 Pending 0 16s
solr-9-zookeeper-1 0/1 ContainerCreating 0 16s
solr-9-solrcloud-2 0/2 Init:1/2 0 17s
solr-9-solrcloud-2 0/2 Init:1/2 0 18s
solr-9-zookeeper-0 1/1 Running 0 22s
solr-9-zookeeper-2 1/1 Running 0 23s
solr-9-zookeeper-1 0/1 Running 0 24s
solr-9-zookeeper-1 1/1 Running 0 34s
solr-9-solrcloud-0 0/2 PodInitializing 0 45s
solr-9-solrcloud-2 0/2 PodInitializing 0 47s
solr-9-solrcloud-0 1/2 Running 0 48s
solr-9-solrcloud-1 0/2 PodInitializing 0 50s
solr-9-solrcloud-2 1/2 Running 0 51s
solr-9-solrcloud-1 1/2 Running 0 55s
solr-9-solrcloud-0 1/2 Running 0 58s
solr-9-solrcloud-0 2/2 Running 0 60s
solr-9-solrcloud-2 1/2 Running 0 63s
solr-9-solrcloud-1 1/2 Running 0 64s
solr-9-solrcloud-2 2/2 Running 0 64s
solr-9-solrcloud-1 2/2 Running 0 66s

.
.
.
##### Zookeeper has empty security.json ###############
I have no name!@solr-9-zookeeper-0:/$ zkCli.sh
/opt/bitnami/java/bin/java
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null zxid: -1
[zk: localhost:2181(CONNECTED) 0] get /solr/security.json
{}

```

Slor Cloud POD's **setup-zk** logs are as below
```
helm/solr-9 % kubectl logs solr-9-solrcloud-0 -c setup-zk
WARN - 2025-04-04 20:21:36.604; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 20:21:38.097; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes

ERROR: KeeperErrorCode = NoNode for /solr

WARN - 2025-04-04 20:21:43.898; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 20:21:45.301; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
Creating ZooKeeper path /solr on ZooKeeper at solr-9-zookeeper-0.solr-9-zookeeper-headless:2181,solr-9-zookeeper-1.solr-9-zookeeper-headless:2181,solr-9-zookeeper-2.solr-9-zookeeper-headless:2181
No security.json found. Put new security.json in ZK
```

I restarted Slor Cloud PODs (scale down the statefulset and let operator scale the PODs back.). Now setup-zk containers say the below:

```
helm/solr-9 % kubectl logs solr-9-solrcloud-0 -c setup-zk
WARN - 2025-04-04 22:14:10.453; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 22:14:12.626; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
configs
overseer
aliases.json
live_nodes
collections
overseer_elect
security.json
node_roles
No security.json found. Put new security.json in ZK
```

So, the only solution is manually add security.json because the operator never creates default one.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce a fresh Helm installation and inspect the Solr Cloud setup-zk container logs, especially the security.json handling shown in the issue. Verify the resulting ZooKeeper state with zkCli.sh and check that the expected security.json is created at /solr rather than requiring manual insertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes
Domain
devops, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.