[Bug] Data skew in Hash Bucketing with CRC32
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
master / 4.0.5
### What's Wrong?
We have a table with 484,862,245,170 rows, and the data is evenly divided according to `shard_num`.
```sql
CREATE TABLE user_label_double_v1
(
`shard_num` SMALLINT NOT NULL
`label_name` VARCHAR(64) NOT NULL
`label_value` DOUBLE NOT NULL
`uid` INT NOT NULL
`dt` Date NOT NULL
)
DUPLICATE KEY(shard_num, label_name)
AUTO PARTITION BY RANGE (date_trunc(`dt`, 'day')) ()
DISTRIBUTED BY HASH(shard_num) BUCKETS 512
PROPERTIES (
"replication_num" = "3",
"compression" = "zstd"
);
```
Define `DISTRIBUTED BY HASH(shard_num) BUCKETS 512` of the table in Doris, but we found there are some empty tablets after importing all data.
```sql
Doris root@6.94.183.233:datamill_test2> select shard_num, count() from user_label_double_v1 group by shard_num order by shard_num;
+-----------+-----------+
| shard_num | count(*) |
+-----------+-----------+
| 0 | 947087198 |
| 1 | 947550670 |
| 2 | 947106194 |
-- ...
| 510 | 947408237 |
| 511 | 946735634 |
+-----------+-----------+
512 rows in set
Time: 46.628s
```
### What You Expected?
Each data bucket is evenly distributed.
### How to Reproduce?
_No response_
### Anything Else?
When defining Hash Bucketing, we want to specify a custom hash function instead of using the fixed `CRC32`.
For example, we can use `crc32(col) % bucket_num`, or use `xxhash64(col) % bucket_num`, or even `col % bucket_num`.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with the reported CREATE TABLE definition and the GROUP BY query, then trace Doris's hash-bucketing behavior for CRC32 and the 512 buckets. No source files or tests are identified; done would require a decided custom-hash design and a reproducible check that bucket distribution is even.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100