ClickHouse / ClickHouse/ClickHouse

Mutation stuck with NOT_ENOUGH_SPACE exception instead of moving to another disk

Open
#48,392 1 comment 0 reactions 0 assignees View on GitHub
comp-mutations external unexpected behaviour
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Describe what's wrong**

MergeTree table uses a multi-disk storage policy. If there is not enough space on one of the disks and a part stored on that disk is mutated, the mutation is stuck instead of moving the part to another disk.

**How to reproduce**

ClickHouse version: 22.12.3.5

Initial storage configuration:
```
---
storage_configuration:
disks:
disk_hdd_1:
path: "/var/lib/clickhouse-hdd1/"
keep_free_space_bytes: 0
disk_hdd_2:
path: "/var/lib/clickhouse-hdd2/"
keep_free_space_bytes: 0
policies:
default:
volumes:
vol_hdd_1:
disk:
- disk_hdd_1
- disk_hdd_2
```

Create MergeTree table with test data:
```
create table default.test_table
(
number UInt64
)
ENGINE = MergeTree
ORDER BY tuple()
as
select number
from numbers_mt(1000000);
```

Make sure the new part is on `disk_hdd_1`:
```
SELECT *
FROM system.parts AS a
WHERE (database = 'default') AND (table = 'test_table')
FORMAT Vertical

Query id: ea3fd07a-378a-4053-a39d-e4419e272521

Row 1:
──────
partition: tuple()
name: all_1_1_0
uuid: 00000000-0000-0000-0000-000000000000
part_type: Compact
active: 1
marks: 123
rows: 1000000
bytes_on_disk: 4007844
data_compressed_bytes: 4004885
data_uncompressed_bytes: 8000000
marks_bytes: 2952
secondary_indices_compressed_bytes: 0
secondary_indices_uncompressed_bytes: 0
secondary_indices_marks_bytes: 0
modification_time: 2023-04-04 14:17:02
remove_time: 1970-01-01 00:00:00
refcount: 1
min_date: 1970-01-01
max_date: 1970-01-01
min_time: 1970-01-01 00:00:00
max_time: 1970-01-01 00:00:00
partition_id: all
min_block_number: 1
max_block_number: 1
level: 0
data_version: 1
primary_key_bytes_in_memory: 0
primary_key_bytes_in_memory_allocated: 0
is_frozen: 0
database: default
table: test_table
engine: MergeTree
disk_name: disk_hdd_1
path: /var/lib/clickhouse-hdd1/store/5d5/5d55bebe-7a05-417b-a225-fbcad1a62321/all_1_1_0/
hash_of_all_files: 372e46e1f4739bdc704cf75faf4a5294
hash_of_uncompressed_files: 62e3b76cf7c4268b243e6f48bf9c7aaf
uncompressed_hash_of_compressed_files: 531372a86bb6bb722d15943e59fffac4
delete_ttl_info_min: 1970-01-01 00:00:00
delete_ttl_info_max: 1970-01-01 00:00:00
move_ttl_info.expression: []
move_ttl_info.min: []
move_ttl_info.max: []
default_compression_codec: LZ4
recompression_ttl_info.expression: []
recompression_ttl_info.min: []
recompression_ttl_info.max: []
group_by_ttl_info.expression: []
group_by_ttl_info.min: []
group_by_ttl_info.max: []
rows_where_ttl_info.expression: []
rows_where_ttl_info.min: []
rows_where_ttl_info.max: []
projections: []
visible: 1
creation_tid: (1,1,'00000000-0000-0000-0000-000000000000')
removal_tid_lock: 0
removal_tid: (0,0,'00000000-0000-0000-0000-000000000000')
creation_csn: 0
removal_csn: 0
has_lightweight_delete: 0
```

Simulation of no free space on `disk_hdd_1`, changing storage configuration with a highly increased `keep_free_space_bytes` parameter:
```
storage_configuration:
disks:
disk_hdd_1:
path: "/var/lib/clickhouse-hdd1/"
keep_free_space_bytes: 275823478784000
disk_hdd_2:
path: "/var/lib/clickhouse-hdd2/"
keep_free_space_bytes: 0
policies:
default:
volumes:
vol_hdd_1:
disk:
- disk_hdd_1
- disk_hdd_2
```

Run `DELETE WHERE` mutation:
```
alter table default.test_table
delete where number = 0;
```

Mutation stuck and the log is full of `NOT_ENOUGH_SPACE` exceptions:
```
SELECT *
FROM system.mutations
WHERE is_done = 0
FORMAT Vertical

Query id: 0678e742-6791-4283-bd4e-d8267e8aa27d

Row 1:
──────
database: default
table: test_table
mutation_id: mutation_3.txt
command: DELETE WHERE number = 0
create_time: 2023-04-04 15:08:19
block_numbers.partition_id: ['']
block_numbers.number: [3]
parts_to_do_names: ['all_1_1_0']
parts_to_do: 1
is_done: 0
latest_failed_part:
latest_fail_time: 1970-01-01 00:00:00
latest_fail_reason:
```

**Expected behavior**

Expected that the part is moved to disk with enough free space instead of stuck mutations.

**Error message and/or stacktrace**

Error message:
```
2023.04.04 15:09:27.677242 [ 1861509 ] {} void DB::BackgroundJobsAssignee::threadFunc(): Code: 243. DB::Exception: Not enough space for mutating part 'all_1_1_0'. (NOT_ENOUGH_SPACE), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked const&, int, bool) @ 0xe750cda in /usr/bin/clickhouse
1. ? @ 0x7eb67ad in /usr/bin/clickhouse
2. DB::CurrentlyMergingPartsTagger::CurrentlyMergingPartsTagger(std::__1::shared_ptr, unsigned long, DB::StorageMergeTree&, std::__1::shared_ptr const&, bool) @ 0x151d4699 in /usr/bin/clickhouse
3. DB::StorageMergeTree::selectPartsToMutate(std::__1::shared_ptr const&, std::__1::basic_string, std::__1::allocator>*, std::__1::shared_ptr&, std::__1::unique_lock&) @ 0x151dffd3 in /usr/bin/clickhouse
4. DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&) @ 0x151e1038 in /usr/bin/clickhouse
5. DB::BackgroundJobsAssignee::threadFunc() @ 0x14d32a27 in /usr/bin/clickhouse
6. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x133abe61 in /usr/bin/clickhouse
7. DB::BackgroundSchedulePool::threadFunction() @ 0x133af42a in /usr/bin/clickhouse
8. ? @ 0x133b034e in /usr/bin/clickhouse
9. ThreadPoolImpl::worker(std::__1::__list_iterator) @ 0xe809f16 in /usr/bin/clickhouse
10. ? @ 0xe80f0e1 in /usr/bin/clickhouse
11. ? @ 0x7fe0d56bf609 in ?
12. clone @ 0x7fe0d55e4133 in ?
(version 22.12.3.5 (official build))
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.