apache / apache/hugegraph

[Question] 十亿点边如何进行快速导入

Open
#2,607 6 comments 1 reaction 1 assignee Claimed by @JackyYangPassion View on GitHub
hbase question
Dominant language
Java
Stars
3.2k
Forks
636
Avg merge
3d 11h
Merged PRs (30d)
14

Description

### Problem Type (问题类型)

None

### Before submit

- [X] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

### Environment (环境信息)

- Server Version: 1.0.0 (Apache Release Version)
- Backend: RocksDB x nodes, HDD or SSD
- OS: xx CPUs, xx G RAM, Ubuntu 2x.x / CentOS 7.x
- Data Size: xx vertices, xx edges

### Your Question (问题描述)

每天的增量数据大概有10亿条左右,底层的存储用的是hbase。 通过loader工具导入的时候,运行一段时间会出现rest server is too busy to write的情况。 尝试着限制了导入的速率(rate_limit.write=10000),和调大了批量写入的线程占比(batch.max_write_ratio=90),这样导入不会报错,但是导入性能会变的很差。 针对大批量数据的导入有什么调优建议吗。 在保证导入不报错的前提下,最大化的提高导入的性能。

### Vertex/Edge example (问题点 / 边数据举例)

_No response_

### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.