apache / apache/sedona

Geostats Functions in Spark Connect

Open
#2,103 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

I don't think the stats functions are compatible with spark connect today. I tried this in spark 3.5:

```
(python) ➜ python git:(graphframes-0.9.0) ✗ export SPARK_REMOTE=local
(python) ➜ python git:(graphframes-0.9.0) ✗ pytest -v tests/stats
```

and every test that wasn't skipped (for checkpointing) gave this kind of _jvm error:

```
self = , name = '_jvm'

def __getattr__(self, name: str) -> Any:
if name in ["_jsc", "_jconf", "_jvm", "_jsparkSession"]:
> raise PySparkAttributeError(
error_class="JVM_ATTRIBUTE_NOT_SUPPORTED", message_parameters={"attr_name": name}
E pyspark.errors.exceptions.base.PySparkAttributeError: [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `_jvm` is not supported in Spark Connect as it depends on the JVM. If you need to use this attribute, do not use Spark Connect when creating your session.

../../../../.local/share/virtualenvs/python-GYLC1Bm8/lib/python3.10/site-packages/pyspark/sql/connect/session.py:692: PySparkAttributeError

```

Contributor guide

Open the contributing guide

Research direction

Start by running `pytest -v tests/stats` with `SPARK_REMOTE=local`, as reported in the issue. Review the stats tests and the `_jvm` failure in Spark Connect; done means the applicable stats tests run successfully without JVM-dependent errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.