[server] Optimize CompletedSnapshotStoreManager memory usage by using relative paths
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
#### Background
The `CompletedSnapshotStoreManager` in Fluss coordinator has significant memory overhead due to storing absolute paths repeatedly for snapshot metadata, which can lead to OOM errors in production clusters with many table buckets.
#### Current Implementation Issues
Currently, snapshot-related paths are stored as absolute paths in multiple places:
1. **`CompletedSnapshot.snapshotLocation`**: Each snapshot stores a complete absolute path
2. **`KvFileHandle.filePath`**: Each KV file stores a complete absolute path
This creates severe memory redundancy:
- For snapshots belonging to the same `TableBucket`, their `snapshotLocation` values differ only in the final snapshot ID, while the base path prefix (e.g., `hdfs://namenode:8020/fluss/kv/db1/table1-100/0/`) is identical across all snapshots
- Each `KvFileHandle` within a snapshot also stores the complete absolute file path, including highly repetitive path prefixes
- In a typical scenario: with a 120-byte base path, 10 retained snapshots per bucket, and 100 files per snapshot, path prefixes alone consume over 100KB of memory per bucket
- When multiplied across thousands of table buckets in a production cluster, this leads to **multi-GB memory overhead and eventual OOM**
#### Memory Impact Analysis
**Before optimization** (absolute paths):
```
Per snapshot:
- snapshotLocation: ~120 bytes (full path)
- 100 KvFileHandles × ~150 bytes each = ~15KB
Total per snapshot: ~15.12KB
Per bucket (10 snapshots): ~151KB
1000 buckets: ~151MB (paths only, excluding other metadata)
10000 buckets: ~1.5GB (paths only)
```
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the Fluss coordinator with CompletedSnapshotStoreManager, then trace CompletedSnapshot.snapshotLocation and KvFileHandle.filePath to understand how snapshot metadata is stored and consumed. The work is done when repeated absolute path prefixes are represented relatively without changing snapshot behavior, and memory usage is reduced for many retained snapshots and table buckets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100