apache / apache/hugegraph-toolchain

[Bug] 边shard不均导致backup很慢

Open
#449 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
121
Forks
125
Avg merge
10d 10h
Merged PRs (30d)
3

Description

### Bug Type (问题类型)

performance (性能下降)

### Before submit

- [X] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

### Environment (环境信息)

- Server Version: 1.0.0 (Apache Release Version)
- Backend: RocksDB x nodes, HDD or SSD
- OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
- Data Size: xx vertices, xx edges

### Expected & Actual behavior (期望与实际表现)

执行边的备份,发现非常慢(点备份很快),命令为:
`bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph backup -t edge --directory ./backup_test`

通过源码发现问题应该是边shard不均匀,导致大部分的线程执行完各自的小分片之后退出,最后只会有1~2个线程在执行较大的分片。虽然配置了多线程,但最后只有一个线程在执行备份,其他线程均退出。

image

也打印了线程池的活跃情况,一开始全部活跃,到最后只有2个活跃。(忽略线程数与备份生成的文件数不一致的情况,修改过线程数)
image

image

### Vertex/Edge example (问题点 / 边数据举例)

_No response_

### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the reported `bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph backup -t edge --directory ./backup_test` command and compare edge and vertex backup behavior. Inspect the backup path and shard workload using the thread-activity evidence in the issue; done means edge backup no longer leaves most threads idle and completes faster.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.