apache / apache/uniffle

[Bug] Incorrect size in `LocalStorageMeta`

Open
#1,247 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

I found some shuffle servers in our cluster write shuffle data to HDFS frequently. And then i found the size stored in `LocalStorageMeta` is incorrect.
Maybe we should update the metrics in the following method. There may be other places that have been missed.
https://github.com/apache/incubator-uniffle/blob/b2154c722f30ee31a867561f550254842c33cbab/server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java#L346

Other suggestion:
1. Maybe it is not a good way that judging if the local disk can be written by using this metric, because this metric is easily overlooked and it is hard to guarantee it's accurate. And #1071 trying to change this.
2. I found `shuffleMetaMap` in `LocalStorageMeta` is useless, maybe we should remove it. And then we can remove `LocalStorageMeta` at the same time.

### Affects Version(s)

master

### Uniffle Server Log Output

_No response_

### Uniffle Engine Log Output

_No response_

### Uniffle Server Configurations

_No response_

### Uniffle Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the referenced update path in server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java around line 346, then inspect LocalStorageMeta and its shuffleMetaMap usage. Trace all places that update the stored size and compare them with the HDFS-writing path; done means the size metric is accurate across those paths and the proposed cleanup is resolved against #1071.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.