apache / apache/doris

[Bug] Data skew in Hash Bucketing with CRC32

Open
#66,231 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

master / 4.0.5

### What's Wrong?

We have a table with 484,862,245,170 rows, and the data is evenly divided according to `shard_num`.

```sql
CREATE TABLE user_label_double_v1
(
`shard_num` SMALLINT NOT NULL
`label_name` VARCHAR(64) NOT NULL
`label_value` DOUBLE NOT NULL
`uid` INT NOT NULL
`dt` Date NOT NULL
)
DUPLICATE KEY(shard_num, label_name)
AUTO PARTITION BY RANGE (date_trunc(`dt`, 'day')) ()
DISTRIBUTED BY HASH(shard_num) BUCKETS 512
PROPERTIES (
"replication_num" = "3",
"compression" = "zstd"
);
```

Define `DISTRIBUTED BY HASH(shard_num) BUCKETS 512` of the table in Doris, but we found there are some empty tablets after importing all data.

```sql
Doris root@6.94.183.233:datamill_test2> select shard_num, count() from user_label_double_v1 group by shard_num order by shard_num;
+-----------+-----------+
| shard_num | count(*) |
+-----------+-----------+
| 0 | 947087198 |
| 1 | 947550670 |
| 2 | 947106194 |
-- ...
| 510 | 947408237 |
| 511 | 946735634 |
+-----------+-----------+
512 rows in set
Time: 46.628s
```

Image

### What You Expected?

Each data bucket is evenly distributed.

### How to Reproduce?

_No response_

### Anything Else?

When defining Hash Bucketing, we want to specify a custom hash function instead of using the fixed `CRC32`.

For example, we can use `crc32(col) % bucket_num`, or use `xxhash64(col) % bucket_num`, or even `col % bucket_num`.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the reported CREATE TABLE definition and the GROUP BY query, then trace Doris's hash-bucketing behavior for CRC32 and the 512 buckets. No source files or tests are identified; done would require a decided custom-hash design and a reproducible check that bucket distribution is even.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.