[Bug] Incorrect size in `LocalStorageMeta`
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
I found some shuffle servers in our cluster write shuffle data to HDFS frequently. And then i found the size stored in `LocalStorageMeta` is incorrect.
Maybe we should update the metrics in the following method. There may be other places that have been missed.
https://github.com/apache/incubator-uniffle/blob/b2154c722f30ee31a867561f550254842c33cbab/server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java#L346
Other suggestion:
1. Maybe it is not a good way that judging if the local disk can be written by using this metric, because this metric is easily overlooked and it is hard to guarantee it's accurate. And #1071 trying to change this.
2. I found `shuffleMetaMap` in `LocalStorageMeta` is useless, maybe we should remove it. And then we can remove `LocalStorageMeta` at the same time.
### Affects Version(s)
master
### Uniffle Server Log Output
_No response_
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start with the referenced update path in server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java around line 346, then inspect LocalStorageMeta and its shuffleMetaMap usage. Trace all places that update the stored size and compare them with the HDFS-writing path; done means the size metric is accurate across those paths and the proposed cleanup is resolved against #1071.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100