apache / apache/arrow-dotnet

Variant shredding cannot express a SQL-null row: the shred pipeline has no validity

Open
#398 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
39
Forks
30
Avg merge
1d 9h
Merged PRs (30d)
16

Description

### Describe the enhancement requested

Every entry point in `Apache.Arrow.Operations.Shredding` works on values, never on a column with validity, so there is no way to shred a nullable variant column:

```csharp
ShredSchema ShredSchemaInferer.Infer(IEnumerable values, ShredOptions options = null)
(byte[], IReadOnlyList) VariantShredder.Shred(IEnumerable values, ShredSchema schema)
VariantArray ShreddedVariantArrayBuilder.Build(ShredSchema schema, byte[] metadata, IReadOnlyList rows, MemoryAllocator allocator = null)
```

`Build` produces a `VariantArray` whose storage struct carries no validity buffer, so every row is valid. A caller shredding a column that contains SQL NULLs has to pass a placeholder value for each null row and then repair the result afterwards.

This matters because the two things are not interchangeable. The parquet VariantShredding spec gives each encoding a distinct meaning: a SQL-NULL row is the optional group itself being null (Arrow: the storage struct's validity), while a present value holding a variant JSON null is `value` = basic type 0 / physical type 0. Collapsing them changes what `IS NULL` means for every consumer of the column, and the round trip stops being lossless.

### Repro

`Apache.Arrow.Operations` 23.0.0 (the only published version), .NET 8:

```csharp
using Apache.Arrow;
using Apache.Arrow.Operations.Shredding;
using Apache.Arrow.Operations.VariantJson;
using Apache.Arrow.Scalars.Variant;

static VariantValue Obj(int a) => VariantValue.FromObject(
new Dictionary { ["a"] = VariantValue.FromInt32(a) });

// Three rows whose MIDDLE row is meant to be SQL NULL. Nothing in the pipeline takes a mask,
// so the best a caller can do is put a placeholder there.
var values = new List { Obj(1), VariantValue.Null, Obj(3) };

var schema = new ShredSchemaInferer().Infer(values, ShredOptions.Default);
var (metadata, rows) = VariantShredder.Shred(values, schema);
VariantArray array = ShreddedVariantArrayBuilder.Build(schema, metadata, rows);

Console.WriteLine($"storage NullCount = {array.StorageArray.NullCount}");
for (int i = 0; i < array.Length; i++)
Console.WriteLine($" row {i}: IsNull={array.IsNull(i),-5} logical={VariantJsonWriter.ToJson(array.GetLogicalVariantValue(i), false)}");
```

```
storage NullCount = 0
row 0: IsNull=False logical={"a":1}
row 1: IsNull=False logical=null <-- wanted a NULL ROW, got a present JSON null
row 2: IsNull=False logical={"a":3}
```

### Current workaround

Rebuild the storage struct with a validity bitmap and re-wrap it, which reaches past the public shredding API into `ArrayData`:

```csharp
var storage = array.StorageArray.Data;
var validity = new ArrowBuffer.BitmapBuilder(array.Length);
validity.Append(true); validity.Append(false); validity.Append(true);
var patched = new VariantArray(array.VariantType, ArrowArrayFactory.BuildArray(
new ArrayData(storage.DataType, storage.Length, nullCount: 1, storage.Offset,
new[] { validity.Build() }, storage.Children, storage.Dictionary)));
// row 1: IsNull=True
```

Two sharp edges in it: the bitmap is built from bit 0 while the `ArrayData` keeps `storage.Offset`, so it is only correct for an unsliced array; and the placeholder still travels through `VariantShredder.Shred`, so it has to be a value the shredder accepts for the inferred schema.

### Suggested API

An overload that carries validity through, e.g.

```csharp
VariantArray ShreddedVariantArrayBuilder.Build(
ShredSchema schema, byte[] metadata, IReadOnlyList rows,
ReadOnlySpan isNull, MemoryAllocator allocator = null);
```

or a validity `ArrowBuffer` + `nullCount` pair if that fits the surrounding style better. Null rows would also be excluded from `ShredSchemaInferer.Infer`, which today has to be done by the caller filtering the sequence.

### Component(s)

C#

Contributor guide

Open the contributing guide

Research direction

Start with the Apache.Arrow.Operations.Shredding entry points named in the issue: ShredSchemaInferer.Infer, VariantShredder.Shred, and ShreddedVariantArrayBuilder.Build; inspect how VariantArray storage validity is represented. Run the supplied .NET 8 repro, then verify that SQL-NULL rows remain null while present variant JSON null values remain valid, including the inference behavior for null rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.