awslabs / awslabs/python-deequ
default value for SPARK_VERSION and spark_session
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
Why can't specifying the Pyspark version in environment variables be optional?
like this
```python
import os
import pyspark
os.environ['SPARK_VERSION'] = str(pyspark.__version__)
```
or
```python
@lru_cache(maxsize=None)
def _get_spark_version() -> str:
try:
spark_version = os.environ.get("SPARK_VERSION")
if not spark_version:
spark_version = str(pyspark.__version__)
except KeyError:
raise RuntimeError(f"SPARK_VERSION environment variable is required. Supported values are: {SPARK_TO_DEEQU_COORD_MAPPING.keys()}")
return _extract_major_minor_versions(spark_version)
```
And default spark version ...
like this
```python
class VerificationResult:
"""The results returned from the VerificationSuite
:param verificationRunBuilder verificationRun: verification result run()
"""
def __init__(self, spark_session: Optional[SparkSession], verificationRun):
self._spark_session = self._setup_spark_session(spark_session)
self.verificationRun = verificationRun
def _setup_spark_session(self, session=None):
if session:
return session
potencial_session = SparkSession.getActiveSession()
if potencial_session:
return potencial_session
else:
msg = "Spark session not found, init with `VerificationResult(my_session, ...)`"
raise AttributeError(msg)
```
maybe i can do this and create PR?
Contributor guide
Assessment
This issue has not been assessed yet.