apache / apache/iceberg

API: Compute variant sizes without full serialization in VariantTestUtil

Open
#17,510 1 comment 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

## Problem

`VariantTestUtil.createArray` pre-computes the data section size by summing `sizeInBytes()` for each element before allocating the output buffer:

```java
// api/src/test/java/org/apache/iceberg/variants/VariantTestUtil.java:273
for (VariantValue value : values) {
// TODO: produce size for every variant without serializing
dataSize += value.sizeInBytes();
}
```

This works for pre-serialized values (`SerializedPrimitive`, `SerializedShortString`, etc.) where `sizeInBytes()` is just `buffer().remaining()`. It breaks down for nested or non-buffer-backed values — e.g. a `SerializedArray` passed as an element may not reflect the size needed when embedded in a parent array without already being fully serialized.

The same pattern exists in `createObject` (line 220) without a TODO, but would benefit from the same fix.

## Proposed behavior

Provide a way to compute the serialized byte size of any `VariantValue` without building the full parent buffer first. Options to consider:

1. Add a `sizeInBytes()` implementation (or helper) that recursively computes size from type/metadata for all variant physical types
2. Add a `VariantTestUtil.sizeInBytes(VariantValue)` helper that handles nested arrays/objects correctly
3. Refactor `createArray` / `createObject` to use it and remove the TODO

`createArray` should then allocate the correct buffer size for mixed and nested element types without redundant serialization work.

## Scope

- **Module:** `api` (test utility — `VariantTestUtil.java`)
- **May touch:** `VariantValue` / serialized variant classes if a shared size API is added
- **Tests:** `TestSerializedArray`, `TestSerializedObject`, and any tests using `VariantTestUtil.createArray` with nested values
- **No production API changes** unless maintainers prefer a public `sizeInBytes` improvement in core variant types

## Notes

- Test-only improvement unless core variant sizing is generalized.
- `createObject` uses the same `sizeInBytes()` loop — consider fixing both together.
- Related tests already cover nested arrays (`TestSerializedArray.testArrayOfMixedTypes`).

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start in api/src/test/java/org/apache/iceberg/variants/VariantTestUtil.java at createArray and createObject, then read the sizeInBytes implementations used by nested VariantValue types. Run TestSerializedArray, TestSerializedObject, and tests using createArray with nested values. Done means mixed and nested arrays and objects allocate the correct size without redundant parent serialization, with the TODO removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.