apache / apache/sedona

GeoPandas: leading-null GeoSeries input loses Z dimensions

Closed
#3,369 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

### Expected behavior

A leading null should not change the dimensions of the other geometries in a GeoSeries. Empty XYZ geometries and points whose Z is NaN should retain their dimensional metadata.

### Actual behavior

The constructor path added in #3349 for #3333 converts local geometries through WKB and ST_GeomFromWKB. Adding a leading null causes these geometries to become 2D:

| Input geometry | GeoSeries([geometry, None]) | GeoSeries([None, geometry]) |
|---|---|---|
| POINT Z EMPTY | 3D | 2D |
| LINESTRING Z EMPTY | 3D | 2D |
| POLYGON Z EMPTY | 3D | 2D |
| POINT Z (1 2 NaN) | 3D | 2D |
| POINT Z (1 2 3) | 3D | 3D |
| POINT EMPTY | 2D | 2D |

### Steps to reproduce

With a SedonaContext initialized using the current-master JVM jar:

```python
import shapely
from sedona.spark.geopandas import GeoSeries

g = shapely.from_wkt("POINT Z EMPTY")
ordinary = GeoSeries([g, None]).to_geopandas().iloc[0]
leading_null = GeoSeries([None, g]).to_geopandas().iloc[1]

print(ordinary.wkt, shapely.get_coordinate_dimension(ordinary))
# POINT Z EMPTY 3
print(leading_null.wkt, shapely.get_coordinate_dimension(leading_null))
# POINT EMPTY 2
```

### Sedona version

2.0.0-SNAPSHOT, master commit 15e415af1df7620a25c2dda43532ef1211c9e9f4.

The Spark 4.1 JVM jar was freshly built from this commit with `mvn -B -pl spark-shaded -am -Dspark=4.1 -Dscala=2.13 -DskipTests package`. Both Python serializer modes reproduced the same results: the pure-Python fallback and a native extension freshly compiled from the same source. The six inputs above were compared through both constructor paths in each mode.

### Environment

- API: Python / GeoPandas compatibility API
- Spark: 4.1.1; Scala: 2.13.17
- Java: 17.0.13; Python: 3.11.13
- Shapely: 2.1.2; GEOS: 3.13.1
- GeoPandas: 1.1.1; pandas: 2.2.3
- Local Spark, macOS arm64

### Root cause and suggested fix

Direct JVM checks show that the WKB parser retains the 3D coordinate sequence. [GeometrySerializer](https://github.com/apache/sedona/blob/15e415af1df7620a25c2dda43532ef1211c9e9f4/common/src/main/java/org/apache/sedona/common/geometrySerde/GeometrySerializer.java#L477-L495) then chooses XY when the sequence has no coordinates or all Z values are NaN. JVM result rows are already 2D before Python decoding.

Replace the [leading-null constructor's WKB conversion](https://github.com/apache/sedona/blob/15e415af1df7620a25c2dda43532ef1211c9e9f4/python/sedona/spark/geopandas/geoseries.py#L740-L754) with construction using an explicit Spark GeometryType schema and Sedona's Python geometry serializer. Preserve the pandas index, row order, name, CRS, embedded SRIDs, and normalization of all pandas missing-value variants. Add dimensional regression cases alongside the existing constructor tests and verify Spark 3.5 and 4.1.

A local explicit-schema prototype preserved the dimensional cases, names, duplicate MultiIndex order, missing values, and CRS on Spark 4.1.1. One additional requirement is to preserve embedded SRIDs in pure-Python serialization: the fallback currently writes zero SRID bytes, so switching to it directly would regress the existing leading-null embedded-SRID case. The native serializer already preserves these SRIDs. A full replacement also needs M/ZM compatibility coverage, since the fallback Python serializer currently rejects measured geometries.

Changing the JVM serializer to treat every three-dimensional JTS sequence as XYZ would also promote ordinary XY data, because JTS can represent XY with a NaN Z slot.

The reproduction measures returned geometry dimensions directly. It does not rely on ST_NDims, ST_CoordDim, or ST_Zmflag, whose empty-geometry results were intentionally changed in #3360.

### Existing issues

- [x] I searched the existing issues and did not find a duplicate.

Contributor guide

Open the contributing guide

Research direction

Run the listed POINT Z EMPTY reproduction in the Python GeoPandas compatibility API, then inspect the leading-null path in python/sedona/spark/geopandas/geoseries.py and GeometrySerializer.java lines 477-495. Add dimensional regression cases alongside the existing constructor tests. Done means preserving XYZ, M/ZM, SRIDs, missing-value handling, index and metadata while verifying Spark 3.5 and 4.1.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, spark
Domain
api, backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.