apache / apache/datafusion

ArrowBytesMap and ArrowBytesViewMap undercount memory by not accounting for initial hash table allocation

Open Beginner friendly
#21,248 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`ArrowBytesMap` and `ArrowBytesViewMap` allocate their hash tables with `HashTable::with_capacity(INITIAL_MAP_CAPACITY)` (128 and 512 entries respectively) but initialize `map_size` to `0`. The `size()` method returns `map_size` as the hash table's contribution to total memory.

The `insert_accounted` method (from `HashTableAllocExt`) only adds to `map_size` when the table needs to grow beyond its current capacity. Since the initial allocation happens in `with_capacity`, not through `insert_accounted`, those bytes are never counted — `size()` understates memory usage until the first resize.

```rust
// binary_map.rs — INITIAL_MAP_CAPACITY = 128
// binary_view_map.rs — INITIAL_MAP_CAPACITY = 512
pub fn new(output_type: OutputType) -> Self {
Self {
map: hashbrown::hash_table::HashTable::with_capacity(INITIAL_MAP_CAPACITY),
map_size: 0, // ← should be map.allocation_size()
// ...
}
}
```

`insert_accounted` only tracks growth:
```rust
fn insert_accounted(&mut self, x: Self::T, hasher: impl Fn(&Self::T) -> u64, accounting: &mut usize) {
if self.len() == self.capacity() {
let bump_size = self.capacity().max(16) * size_of::();
*accounting = (*accounting).checked_add(bump_size).expect("overflow");
self.reserve(bump_elements, &hasher);
}
self.insert_unique(hash, x, hasher);
}
```

### To Reproduce

N/A

### Expected behavior

`map_size` should be initialized with `map.allocation_size()` to account for the pre-allocated memory from `with_capacity`.

### Additional context

Two other uses of `map_size: 0` in the codebase (`row.rs` and `multi_group_by/mod.rs`) use `HashTable::with_capacity(0)` which allocates nothing, so they are already correct.

Contributor guide

Open the contributing guide

Research direction

Start by reading binary_map.rs and binary_view_map.rs, focusing on each map's with_capacity initialization and size accounting. Compare the zero-capacity HashTable uses in row.rs and multi_group_by/mod.rs, then verify that the reported sizes include the initial allocations for both map types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.