Variant shredding has no array-level entry points: every consumer re-writes the same VariantArray loops
- Dominant language
- C#
- Stars
- 39
- Forks
- 30
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 16
Description
### Describe the enhancement requested
`Apache.Arrow.Operations.Shredding` implements the parquet VariantShredding spec one value at a time, but a shredded variant is a COLUMN-level layout. There is no entry point in either direction that takes or returns a `VariantArray`:
- **read**: `VariantArrayShreddingExtensions.GetLogicalVariantValue(VariantArray, int)` resolves one row's typed_value + residual, and `VariantUnshredder.Reconstruct` works on a single `ShredResult`. Nothing turns a shredded `VariantArray` into its canonical unshredded equivalent.
- **write**: `ShredSchemaInferer.Infer` / `VariantShredder.Shred` / `ShreddedVariantArrayBuilder.Build` take and return values and `ShredResult`s. Nothing takes a `VariantArray`.
So every consumer that holds a column writes the same two loops. Ours, from a parquet implementation that reassembles on read and shreds on write:
```csharp
public static VariantArray Reassemble(VariantArray array)
{
if (!array.IsShredded) return array;
var builder = new VariantArray.Builder();
for (int i = 0; i < array.Length; i++)
{
if (array.IsNull(i)) { builder.AppendNull(); continue; }
builder.Append(array.GetLogicalVariantValue(i));
}
return builder.Build(allocator: null);
}
```
and, on the way in, a loop that decodes every row to `VariantValue` (plus a null mask) purely to hand the sequence to `Infer`/`Shred`.
Neither loop encodes any judgement — they are the obvious implementation, which is the argument for having them once, upstream, next to the code whose invariants they depend on. As it stands each consumer re-derives details like "read the LOGICAL value so an already-shredded input re-shreds rather than losing its typed columns", and gets to find out the hard way that `VariantArray` is not an `Apache.Arrow.Array`.
### Suggested API
```csharp
// read
public static VariantArray Reassemble(this VariantArray array);
// write
public static VariantArray Shred(this VariantArray array, ShredSchema schema);
public static bool TryShred(this VariantArray array, ShredOptions options, out VariantArray shredded);
```
Splitting inference from shredding matters for the write side, and is why `Shred(array, schema)` is listed separately from `TryShred`: a parquet file has ONE schema while a writer sees one batch at a time, so a caller must infer once over a representative batch and shred every later batch into that same layout. An API that only offered infer-and-shred-together would quietly produce row groups whose layouts disagree.
These also compose with #398 — an array-level entry point has somewhere natural to carry validity, which the value-level one does not.
Happy to open a PR for any of this if the shape sounds right.
### Component(s)
C#
Contributor guide
Research direction
Start in Apache.Arrow.Operations.Shredding by reading VariantArrayShreddingExtensions.GetLogicalVariantValue and the value-level VariantUnshredder.Reconstruct path, then inspect ShredSchemaInferer.Infer, VariantShredder.Shred, and ShreddedVariantArrayBuilder.Build. Define and validate array-level read and write entry points that preserve nulls, reassemble logical values, and keep schema inference separate from shredding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend-api-design, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100