apache / apache/iceberg

Spark: Support nested geospatial values in StructInternalRow for older versions

Open
#17,720 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Problem

`StructInternalRow.collectionToArrayData` does not handle Iceberg `GEOMETRY` or `GEOGRAPHY` elements in older Spark modules. In Spark 4.1, nested geospatial values in arrays and maps reach the default switch branch and throw:

```text
UnsupportedOperationException: Unsupported array element type
```

PR #14984 adds nested conversion in the Spark 4.2 module while adapting to Spark 4.2's `BinaryView` physical representation. The nested array/map support is new capability rather than a Spark 4.2 compatibility requirement and should be evaluated and ported independently to applicable older Spark versions.

Spark 4.1 uses `GeometryVal` and `GeographyVal`, so its implementation must construct those values rather than copying the Spark 4.2 `BinaryView` code. Spark 4.0 applicability should be confirmed against that module's geospatial type mapping before adding code.

The top-level generic geospatial getter in Spark 4.2 is a separate correctness fix and should remain in #14984.

### Proposed fix

- Add `GEOMETRY` and `GEOGRAPHY` handling to `StructInternalRow.collectionToArrayData` in supported older Spark modules.
- Convert both `ByteBuffer` and `byte[]` Iceberg WKB inputs into the Spark-version-specific physical value.
- Cover geometry and geography in arrays and as map keys/values.
- Cover null collection values and nested combinations such as a struct inside a list.
- Include representative non-point WKB and relevant SRID behavior.

### Context

Identified while reviewing Spark 4.2 support in #14984: https://github.com/apache/iceberg/pull/14984#discussion_r3788259724

Contributor guide

Open the contributing guide

Research direction

Start with StructInternalRow.collectionToArrayData in the older Spark modules and compare the nested conversion added by PR #14984 for Spark 4.2. Confirm Spark 4.0's geospatial type mapping before deciding whether it applies, then trace Spark 4.1's GeometryVal and GeographyVal representations. Done means supported modules handle ByteBuffer and byte[] GEOMETRY/GEOGRAPHY values in arrays and map keys or values, including nulls and nested structs, with representative WKB and SRID tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.