apache / apache/linkis

[Feature][PublicService] Optimize OOM issue in ps-public service after enabling Kerberos authentication

Open
#5,312 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.4k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

### Linkis Component

linkis-public-enhancements/linkis-ps-public-service

### What happened

**English:**

After enabling Kerberos authentication, the ps-public service frequently encounters Out Of Memory (OOM) issues due to excessive memory consumption in authentication-related operations.

**Problem Description:**
When Kerberos authentication is enabled, the ps-public service experiences significant memory pressure and eventually runs out of memory. This appears to be related to Kerberos ticket caching, principal object creation, or authentication state management.

---

**中文:**

开启Kerberos认证后,ps-public服务由于认证相关操作的内存消耗过大,频繁出现内存溢出(OOM)问题。

**问题描述:**
启用Kerberos认证时,ps-public服务会遇到严重的内存压力并最终耗尽内存。这似乎与Kerberos票据缓存、主体对象创建或认证状态管理有关。

### What you expected to happen

**English:**

After enabling Kerberos authentication, the ps-public service should:
1. Maintain stable memory usage without OOM
2. Properly cache and reuse Kerberos tickets
3. Release authentication objects after use
4. Implement memory-efficient Kerberos authentication

---

**中文:**

开启Kerberos认证后,ps-public服务应该:
1. 保持稳定的内存使用而不会OOM
2. 正确缓存和重用Kerberos票据
3. 使用后释放认证对象
4. 实现内存高效的Kerberos认证

### How to reproduce

**English:**
1. Enable Kerberos authentication in Linkis configuration
2. Configure ps-public service with Kerberos principals
3. Submit multiple tasks or requests through ps-public
4. Monitor service memory usage over time
5. Observe gradual memory increase leading to OOM

---

**中文:**
1. 在Linkis配置中启用Kerberos认证
2. 使用Kerberos主体配置ps-public服务
3. 通过ps-public提交多个任务或请求
4. 监控服务内存使用随时间的变化
5. 观察逐渐增加的内存导致OOM

### Anything else

**English:**

**Potential Root Causes:**
1. **Kerberos ticket cache not properly managed**: Tickets accumulate without cleanup
2. **Principal objects not released**: UserGroupInformation or LoginContext objects retained
3. **Memory leak in authentication interceptor**: Objects created per request not garbage collected
4. **Excessive ticket renewal**: Too frequent renewal operations consuming memory

**Suggested Solutions:**
1. **Implement ticket cache management**: Add LRU cache with size limits for Kerberos tickets
2. **Use object pooling**: Reuse LoginContext and UGI objects instead of creating new ones
3. **Add cleanup hooks**: Ensure proper cleanup of Kerberos objects after authentication
4. **Optimize renewal strategy**: Reduce ticket renewal frequency, implement lazy renewal
5. **Memory profiling**: Use heap dump analysis to identify specific leak sources

**Configuration Recommendations:**
```properties
# JVM settings for Kerberos-enabled service
-Xms4g -Xmx8g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-Djava.security.krb5.conf=/etc/krb5.conf
-Dsun.security.krb5.debug=false # Disable Kerberos debug logging in production
```

---

**中文:**

**可能的根本原因:**
1. **Kerberos票据缓存未正确管理**:票据累积而没有清理
2. **主体对象未释放**:UserGroupInformation或LoginContext对象被保留
3. **认证拦截器中的内存泄漏**:每个请求创建的对象未被垃圾回收
4. **票据更新过于频繁**:过于频繁的更新操作消耗内存

**建议解决方案:**
1. **实现票据缓存管理**:为Kerberos票据添加具有大小限制的LRU缓存
2. **使用对象池**:重用LoginContext和UGI对象而不是创建新对象
3. **添加清理钩子**:确保认证后正确清理Kerberos对象
4. **优化更新策略**:减少票据更新频率,实现延迟更新
5. **内存分析**:使用堆转储分析识别具体的泄漏源

**配置建议:**
```properties
# 启用Kerberos的服务的JVM设置
-Xms4g -Xmx8g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-Djava.security.krb5.conf=/etc/krb5.conf
-Dsun.security.krb5.debug=false # 生产环境禁用Kerberos调试日志
```

### Are you willing to submit a PR?


- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the linkis-public-enhancements/linkis-ps-public-service component and reproduce the memory increase with Kerberos enabled while monitoring service usage. Use heap dump analysis to determine whether ticket caches, UserGroupInformation or LoginContext objects, authentication interceptors, or renewal operations retain memory. Done means stable memory under repeated requests without OOM and appropriate ticket reuse and object cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
authentication, backend, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.