API: Compute variant sizes without full serialization in VariantTestUtil
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
## Problem
`VariantTestUtil.createArray` pre-computes the data section size by summing `sizeInBytes()` for each element before allocating the output buffer:
```java
// api/src/test/java/org/apache/iceberg/variants/VariantTestUtil.java:273
for (VariantValue value : values) {
// TODO: produce size for every variant without serializing
dataSize += value.sizeInBytes();
}
```
This works for pre-serialized values (`SerializedPrimitive`, `SerializedShortString`, etc.) where `sizeInBytes()` is just `buffer().remaining()`. It breaks down for nested or non-buffer-backed values — e.g. a `SerializedArray` passed as an element may not reflect the size needed when embedded in a parent array without already being fully serialized.
The same pattern exists in `createObject` (line 220) without a TODO, but would benefit from the same fix.
## Proposed behavior
Provide a way to compute the serialized byte size of any `VariantValue` without building the full parent buffer first. Options to consider:
1. Add a `sizeInBytes()` implementation (or helper) that recursively computes size from type/metadata for all variant physical types
2. Add a `VariantTestUtil.sizeInBytes(VariantValue)` helper that handles nested arrays/objects correctly
3. Refactor `createArray` / `createObject` to use it and remove the TODO
`createArray` should then allocate the correct buffer size for mixed and nested element types without redundant serialization work.
## Scope
- **Module:** `api` (test utility — `VariantTestUtil.java`)
- **May touch:** `VariantValue` / serialized variant classes if a shared size API is added
- **Tests:** `TestSerializedArray`, `TestSerializedObject`, and any tests using `VariantTestUtil.createArray` with nested values
- **No production API changes** unless maintainers prefer a public `sizeInBytes` improvement in core variant types
## Notes
- Test-only improvement unless core variant sizing is generalized.
- `createObject` uses the same `sizeInBytes()` loop — consider fixing both together.
- Related tests already cover nested arrays (`TestSerializedArray.testArrayOfMixedTypes`).
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start in api/src/test/java/org/apache/iceberg/variants/VariantTestUtil.java at createArray and createObject, then read the sizeInBytes implementations used by nested VariantValue types. Run TestSerializedArray, TestSerializedObject, and tests using createArray with nested values. Done means mixed and nested arrays and objects allocate the correct size without redundant parent serialization, with the TODO removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100