apache / apache/parquet-java

Avoid per-write virtual dispatch in `DictionaryValuesWriter.shouldFallBack()` by caching the size-exceeded check

Open
#3,501 0 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

### Describe the enhancement requested

`DictionaryValuesWriter.shouldFallBack()` is called by `FallbackValuesWriter.checkFallback()` after every single value write. The current implementation dispatches a virtual call to `getDictionarySize()` on every invocation:
```java
public boolean shouldFallBack() {
return dictionaryByteSize > maxDictionaryByteSize || getDictionarySize() > MAX_DICTIONARY_ENTRIES;
}
```
`getDictionarySize()` is an abstract method overridden in each typed subclass (Binary, Long, Double, Integer, Float) to return the backing map's `.size()`. Since `shouldFallBack()` is polled after every write, including writes of duplicate values that do not grow the dictionary, the virtual dispatch and map-size query are redundant work for the common case where most values are already in the dictionary.
Both `dictionaryByteSize` and the dictionary entry count can only increase when a new entry is added (inside the `if (id == -1)` branch of each subclass's write method). Therefore the size-exceeded condition can only transition from `false` to `true` at that exact point.

### Proposal
Replace the per-write check with a cached boolean `dictionarySizeExceeded` flag. Introduce a `checkDictionarySizeLimit(int newDictionarySize)` method that subclass write methods call only when a new dictionary entry is actually added. `shouldFallBack()` then returns the cached flag directly, a simple field read with no virtual dispatch.

### Component(s)

Core

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with DictionaryValuesWriter.shouldFallBack() and FallbackValuesWriter.checkFallback(), then inspect the Binary, Long, Double, Integer, and Float typed subclass write methods, especially their id == -1 branches. Confirm that the size-exceeded state is updated only when a new dictionary entry is added and that fallback behavior remains unchanged for duplicate writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.