apache / apache/arrow-dotnet

Variant shredding has no array-level entry points: every consumer re-writes the same VariantArray loops

Open
#399 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
39
Forks
30
Avg merge
1d 9h
Merged PRs (30d)
16

Description

### Describe the enhancement requested

`Apache.Arrow.Operations.Shredding` implements the parquet VariantShredding spec one value at a time, but a shredded variant is a COLUMN-level layout. There is no entry point in either direction that takes or returns a `VariantArray`:

- **read**: `VariantArrayShreddingExtensions.GetLogicalVariantValue(VariantArray, int)` resolves one row's typed_value + residual, and `VariantUnshredder.Reconstruct` works on a single `ShredResult`. Nothing turns a shredded `VariantArray` into its canonical unshredded equivalent.
- **write**: `ShredSchemaInferer.Infer` / `VariantShredder.Shred` / `ShreddedVariantArrayBuilder.Build` take and return values and `ShredResult`s. Nothing takes a `VariantArray`.

So every consumer that holds a column writes the same two loops. Ours, from a parquet implementation that reassembles on read and shreds on write:

```csharp
public static VariantArray Reassemble(VariantArray array)
{
if (!array.IsShredded) return array;

var builder = new VariantArray.Builder();
for (int i = 0; i < array.Length; i++)
{
if (array.IsNull(i)) { builder.AppendNull(); continue; }
builder.Append(array.GetLogicalVariantValue(i));
}
return builder.Build(allocator: null);
}
```

and, on the way in, a loop that decodes every row to `VariantValue` (plus a null mask) purely to hand the sequence to `Infer`/`Shred`.

Neither loop encodes any judgement — they are the obvious implementation, which is the argument for having them once, upstream, next to the code whose invariants they depend on. As it stands each consumer re-derives details like "read the LOGICAL value so an already-shredded input re-shreds rather than losing its typed columns", and gets to find out the hard way that `VariantArray` is not an `Apache.Arrow.Array`.

### Suggested API

```csharp
// read
public static VariantArray Reassemble(this VariantArray array);

// write
public static VariantArray Shred(this VariantArray array, ShredSchema schema);
public static bool TryShred(this VariantArray array, ShredOptions options, out VariantArray shredded);
```

Splitting inference from shredding matters for the write side, and is why `Shred(array, schema)` is listed separately from `TryShred`: a parquet file has ONE schema while a writer sees one batch at a time, so a caller must infer once over a representative batch and shred every later batch into that same layout. An API that only offered infer-and-shred-together would quietly produce row groups whose layouts disagree.

These also compose with #398 — an array-level entry point has somewhere natural to carry validity, which the value-level one does not.

Happy to open a PR for any of this if the shape sounds right.

### Component(s)

C#

Contributor guide

Open the contributing guide

Research direction

Start in Apache.Arrow.Operations.Shredding by reading VariantArrayShreddingExtensions.GetLogicalVariantValue and the value-level VariantUnshredder.Reconstruct path, then inspect ShredSchemaInferer.Infer, VariantShredder.Shred, and ShreddedVariantArrayBuilder.Build. Define and validate array-level read and write entry points that preserve nulls, reassemble logical values, and keep schema inference separate from shredding.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.