GeoPandas: leading-null GeoSeries input loses Z dimensions
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
### Expected behavior
A leading null should not change the dimensions of the other geometries in a GeoSeries. Empty XYZ geometries and points whose Z is NaN should retain their dimensional metadata.
### Actual behavior
The constructor path added in #3349 for #3333 converts local geometries through WKB and ST_GeomFromWKB. Adding a leading null causes these geometries to become 2D:
| Input geometry | GeoSeries([geometry, None]) | GeoSeries([None, geometry]) |
|---|---|---|
| POINT Z EMPTY | 3D | 2D |
| LINESTRING Z EMPTY | 3D | 2D |
| POLYGON Z EMPTY | 3D | 2D |
| POINT Z (1 2 NaN) | 3D | 2D |
| POINT Z (1 2 3) | 3D | 3D |
| POINT EMPTY | 2D | 2D |
### Steps to reproduce
With a SedonaContext initialized using the current-master JVM jar:
```python
import shapely
from sedona.spark.geopandas import GeoSeries
g = shapely.from_wkt("POINT Z EMPTY")
ordinary = GeoSeries([g, None]).to_geopandas().iloc[0]
leading_null = GeoSeries([None, g]).to_geopandas().iloc[1]
print(ordinary.wkt, shapely.get_coordinate_dimension(ordinary))
# POINT Z EMPTY 3
print(leading_null.wkt, shapely.get_coordinate_dimension(leading_null))
# POINT EMPTY 2
```
### Sedona version
2.0.0-SNAPSHOT, master commit 15e415af1df7620a25c2dda43532ef1211c9e9f4.
The Spark 4.1 JVM jar was freshly built from this commit with `mvn -B -pl spark-shaded -am -Dspark=4.1 -Dscala=2.13 -DskipTests package`. Both Python serializer modes reproduced the same results: the pure-Python fallback and a native extension freshly compiled from the same source. The six inputs above were compared through both constructor paths in each mode.
### Environment
- API: Python / GeoPandas compatibility API
- Spark: 4.1.1; Scala: 2.13.17
- Java: 17.0.13; Python: 3.11.13
- Shapely: 2.1.2; GEOS: 3.13.1
- GeoPandas: 1.1.1; pandas: 2.2.3
- Local Spark, macOS arm64
### Root cause and suggested fix
Direct JVM checks show that the WKB parser retains the 3D coordinate sequence. [GeometrySerializer](https://github.com/apache/sedona/blob/15e415af1df7620a25c2dda43532ef1211c9e9f4/common/src/main/java/org/apache/sedona/common/geometrySerde/GeometrySerializer.java#L477-L495) then chooses XY when the sequence has no coordinates or all Z values are NaN. JVM result rows are already 2D before Python decoding.
Replace the [leading-null constructor's WKB conversion](https://github.com/apache/sedona/blob/15e415af1df7620a25c2dda43532ef1211c9e9f4/python/sedona/spark/geopandas/geoseries.py#L740-L754) with construction using an explicit Spark GeometryType schema and Sedona's Python geometry serializer. Preserve the pandas index, row order, name, CRS, embedded SRIDs, and normalization of all pandas missing-value variants. Add dimensional regression cases alongside the existing constructor tests and verify Spark 3.5 and 4.1.
A local explicit-schema prototype preserved the dimensional cases, names, duplicate MultiIndex order, missing values, and CRS on Spark 4.1.1. One additional requirement is to preserve embedded SRIDs in pure-Python serialization: the fallback currently writes zero SRID bytes, so switching to it directly would regress the existing leading-null embedded-SRID case. The native serializer already preserves these SRIDs. A full replacement also needs M/ZM compatibility coverage, since the fallback Python serializer currently rejects measured geometries.
Changing the JVM serializer to treat every three-dimensional JTS sequence as XYZ would also promote ordinary XY data, because JTS can represent XY with a NaN Z slot.
The reproduction measures returned geometry dimensions directly. It does not rely on ST_NDims, ST_CoordDim, or ST_Zmflag, whose empty-geometry results were intentionally changed in #3360.
### Existing issues
- [x] I searched the existing issues and did not find a duplicate.
Contributor guide
Research direction
Run the listed POINT Z EMPTY reproduction in the Python GeoPandas compatibility API, then inspect the leading-null path in python/sedona/spark/geopandas/geoseries.py and GeometrySerializer.java lines 477-495. Add dimensional regression cases alongside the existing constructor tests. Done means preserving XYZ, M/ZM, SRIDs, missing-value handling, index and metadata while verifying Spark 3.5 and 4.1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, spark
- Domain
- api, backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100