apache / apache/incubator-pegasus
Internal authentication of the cluster failed via Pegasus 2.4
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## General Question
when I use Pegasus 2.4 access controller , I can use peagsus-shell access target cluster , but I found all the RPC will be failed in the internal of the cluster.
1. I use `pegasus_prc/pegasus@COMPANY.HADOOP` as my server principal.
Here is my keytab Principal used in target cluster(To check my keytab file is consistent with principal):
```
[work@xxxxxxxx pegasus]$ klist -k pegasus_prc@COMPANY.HADOOP.keytab
Keytab name: FILE:pegasus_prc@COMPANY.HADOOP.keytab
KVNO Principal
---- --------------------------------------------------------------------------
1 pegasus_prc/pegasus@COMPANY.HADOOP
1 pegasus_prc/pegasus@COMPANY.HADOOP
```
2. Here is the config in target cluster's config.ini:
```
[[security]]
enable_acl = false
super_users = u_guoningshen
service_name = pegasus_prc
service_fqdn = pegasus
sasl_plugin_path = /usr/lib64/sasl2
krb5_keytab = /home/work/app/pegasus/pegasus_prc@COMPANY.HADOOP.keytab
krb5_config = /home/work/app/pegasus/krb5.conf
krb5_principal = pegasus_prc/pegasus@COMPANY.HADOOP
mandatory_auth = false
enable_auth = true
```
3. Here is my pegasus-shell ini file , and I use it to access target cluster
```
[apps..default]
run = true
count = 1
[apps.mimic]
type = dsn.app.mimic
arguments =
pools = THREAD_POOL_DEFAULT,THREAD_POOL_META_SERVER
run = true
count = 1
[core]
tool = nativerun
pause_on_start = false
logging_start_level = LOG_LEVEL_DEBUG
logging_factory_name = dsn::tools::simple_logger
logging_flush_on_exit = false
enable_default_app_mimic = true
data_dir = ./pegasus_shell.data
[tools.simple_logger]
short_header = false
fast_flush = true
max_number_of_log_files_on_disk = 10
stderr_start_level = LOG_LEVEL_FATAL
[tools.simulator]
random_seed = 0
[network]
io_service_worker_count = 4
[threadpool..default]
worker_count = 4
partitioned = false
worker_priority = THREAD_xPRIORITY_NORMAL
[threadpool.THREAD_POOL_DEFAULT]
name = default
worker_count = 20
[threadpool.THREAD_POOL_META_SERVER]
name = meta_server
[task..default]
is_trace = false
is_profile = false
allow_inline = false
rpc_call_header_format = NET_HDR_DSN
rpc_call_channel = RPC_CHANNEL_TCP
rpc_timeout_milliseconds = 10000
[pegasus.clusters]
c4tst-function2 = 10.xxx.xx.1:32601,10.xxx.xx.2:32601
[security]
enable_auth = true
krb5_keytab = /home/work/2.4.4_pegasus/pegasus/u_guoningshen.keytab
krb5_config = /etc/krb5.conf
krb5_principal = u_guoningshen@COMPANY.HADOOP
sasl_plugin_path = /home/work/2.4.4_pegasus/pegasus/thirdparty/output/lib/sasl2
service_fqdn = pegasus
service_name = pegasus_prc
```
4. What happen?
* Connected to cluster via pegasus-shell
```
./run.sh shell -c ker.ini
```
* `u_guoningshen` is super user for cluster , so I have full permissions.
```
The cluster name is: c4tst-function2
The cluster meta list is: 10.xxx.xx.1:32601,10.xxx.xx.2:32601
>>> ls
[general_info]
app_id status app_name app_type partition_count replica_count is_stateful create_time drop_time drop_expire envs_count
238 AVAILABLE test pegasus 4 3 true 2024-09-11_07:30:20 - - 0
239 AVAILABLE gns pegasus 4 3 true 2024-09-12_02:30:50 - - 0
[summary]
total_app_count : 2
>>> drop gns
reserve_seconds = 0
drop app gns succeed
>>> ls
[general_info]
app_id status app_name app_type partition_count replica_count is_stateful create_time drop_time drop_expire envs_count
238 AVAILABLE test pegasus 4 3 true 2024-09-11_07:30:20 - - 0
[summary]
total_app_count : 1
>>>
```
* But I can not create table, because all the rpc send from master meta to another nodes will `negotiation failed, with err = ERR_UNKNOWN, msg = ERR_UNKNOWN`
```
>>> create gns_test
create app gns_test succeed, waiting for app ready
gns_test not ready yet, still waiting... (0/4)
gns_test not ready yet, still waiting... (0/4)
gns_test not ready yet, still waiting... (0/4)
gns_test not ready yet, still waiting... (0/4)
...
```
Contributor guide
Research direction
Reproduce the failure with `./run.sh shell -c ker.ini`, then compare the target cluster's `config.ini` with the shell's `[security]` settings. Trace the internal RPC negotiation from the master meta server to other nodes. Done means table creation completes and the application reaches ready status without negotiation failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- authentication, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100