Variant shredding cannot express a SQL-null row: the shred pipeline has no validity
- Dominant language
- C#
- Stars
- 39
- Forks
- 30
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 16
Description
### Describe the enhancement requested
Every entry point in `Apache.Arrow.Operations.Shredding` works on values, never on a column with validity, so there is no way to shred a nullable variant column:
```csharp
ShredSchema ShredSchemaInferer.Infer(IEnumerable values, ShredOptions options = null)
(byte[], IReadOnlyList) VariantShredder.Shred(IEnumerable values, ShredSchema schema)
VariantArray ShreddedVariantArrayBuilder.Build(ShredSchema schema, byte[] metadata, IReadOnlyList rows, MemoryAllocator allocator = null)
```
`Build` produces a `VariantArray` whose storage struct carries no validity buffer, so every row is valid. A caller shredding a column that contains SQL NULLs has to pass a placeholder value for each null row and then repair the result afterwards.
This matters because the two things are not interchangeable. The parquet VariantShredding spec gives each encoding a distinct meaning: a SQL-NULL row is the optional group itself being null (Arrow: the storage struct's validity), while a present value holding a variant JSON null is `value` = basic type 0 / physical type 0. Collapsing them changes what `IS NULL` means for every consumer of the column, and the round trip stops being lossless.
### Repro
`Apache.Arrow.Operations` 23.0.0 (the only published version), .NET 8:
```csharp
using Apache.Arrow;
using Apache.Arrow.Operations.Shredding;
using Apache.Arrow.Operations.VariantJson;
using Apache.Arrow.Scalars.Variant;
static VariantValue Obj(int a) => VariantValue.FromObject(
new Dictionary { ["a"] = VariantValue.FromInt32(a) });
// Three rows whose MIDDLE row is meant to be SQL NULL. Nothing in the pipeline takes a mask,
// so the best a caller can do is put a placeholder there.
var values = new List { Obj(1), VariantValue.Null, Obj(3) };
var schema = new ShredSchemaInferer().Infer(values, ShredOptions.Default);
var (metadata, rows) = VariantShredder.Shred(values, schema);
VariantArray array = ShreddedVariantArrayBuilder.Build(schema, metadata, rows);
Console.WriteLine($"storage NullCount = {array.StorageArray.NullCount}");
for (int i = 0; i < array.Length; i++)
Console.WriteLine($" row {i}: IsNull={array.IsNull(i),-5} logical={VariantJsonWriter.ToJson(array.GetLogicalVariantValue(i), false)}");
```
```
storage NullCount = 0
row 0: IsNull=False logical={"a":1}
row 1: IsNull=False logical=null <-- wanted a NULL ROW, got a present JSON null
row 2: IsNull=False logical={"a":3}
```
### Current workaround
Rebuild the storage struct with a validity bitmap and re-wrap it, which reaches past the public shredding API into `ArrayData`:
```csharp
var storage = array.StorageArray.Data;
var validity = new ArrowBuffer.BitmapBuilder(array.Length);
validity.Append(true); validity.Append(false); validity.Append(true);
var patched = new VariantArray(array.VariantType, ArrowArrayFactory.BuildArray(
new ArrayData(storage.DataType, storage.Length, nullCount: 1, storage.Offset,
new[] { validity.Build() }, storage.Children, storage.Dictionary)));
// row 1: IsNull=True
```
Two sharp edges in it: the bitmap is built from bit 0 while the `ArrayData` keeps `storage.Offset`, so it is only correct for an unsliced array; and the placeholder still travels through `VariantShredder.Shred`, so it has to be a value the shredder accepts for the inferred schema.
### Suggested API
An overload that carries validity through, e.g.
```csharp
VariantArray ShreddedVariantArrayBuilder.Build(
ShredSchema schema, byte[] metadata, IReadOnlyList rows,
ReadOnlySpan isNull, MemoryAllocator allocator = null);
```
or a validity `ArrowBuffer` + `nullCount` pair if that fits the surrounding style better. Null rows would also be excluded from `ShredSchemaInferer.Infer`, which today has to be done by the caller filtering the sequence.
### Component(s)
C#
Contributor guide
Research direction
Start with the Apache.Arrow.Operations.Shredding entry points named in the issue: ShredSchemaInferer.Infer, VariantShredder.Shred, and ShreddedVariantArrayBuilder.Build; inspect how VariantArray storage validity is represented. Run the supplied .NET 8 repro, then verify that SQL-NULL rows remain null while present variant JSON null values remain valid, including the inference behavior for null rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100