awslabs / awslabs/python-deequ

spark_version not capturing from OS environ in Azure databricks cluster

Open
#143 3 comments 0 reactions 0 assignees View on GitHub
bug enhancement
Dominant language
Jupyter Notebook
Stars
826
Forks
158
Avg merge
9d 22h
Merged PRs (30d)
3

Description

**Describe the bug**
spark_version not capturing from OS environ in Azure databricks cluster.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to 'Azure databricks workspace'
2. Click on 'notebook'
3. Scroll down to 'try to run the pydeequ as a job'
4. See error

![image](https://github.com/awslabs/python-deequ/assets/36189591/c40900fd-49b6-4620-83cb-af2b0433c163)

**Expected behavior**
We can use spark session to retrieve spark version.
![image](https://github.com/awslabs/python-deequ/assets/36189591/4ce0087a-bdb0-4688-8c51-974b9cd71adc)

**Screenshots**

**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]

**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.