Geostats Functions in Spark Connect
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
I don't think the stats functions are compatible with spark connect today. I tried this in spark 3.5:
```
(python) ➜ python git:(graphframes-0.9.0) ✗ export SPARK_REMOTE=local
(python) ➜ python git:(graphframes-0.9.0) ✗ pytest -v tests/stats
```
and every test that wasn't skipped (for checkpointing) gave this kind of _jvm error:
```
self = , name = '_jvm'
def __getattr__(self, name: str) -> Any:
if name in ["_jsc", "_jconf", "_jvm", "_jsparkSession"]:
> raise PySparkAttributeError(
error_class="JVM_ATTRIBUTE_NOT_SUPPORTED", message_parameters={"attr_name": name}
E pyspark.errors.exceptions.base.PySparkAttributeError: [JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `_jvm` is not supported in Spark Connect as it depends on the JVM. If you need to use this attribute, do not use Spark Connect when creating your session.
../../../../.local/share/virtualenvs/python-GYLC1Bm8/lib/python3.10/site-packages/pyspark/sql/connect/session.py:692: PySparkAttributeError
```
Contributor guide
Research direction
Start by running `pytest -v tests/stats` with `SPARK_REMOTE=local`, as reported in the issue. Review the stats tests and the `_jvm` failure in Spark Connect; done means the applicable stats tests run successfully without JVM-dependent errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100