awslabs / awslabs/python-deequ
pydeequ on Azure Databricks running profiler getting keyerror
- Vorherrschende Sprache
- Jupyter Notebook
- Sterne
- 826
- Forks
- 158
- Ø Merge
- 9 T. 22 Std.
- Gemergte PRs (30 T.)
- 3
Beschreibung
Code:
from pydeequ.profiles import *
result = ColumnProfilerRunner(spark) \
.onData(df) \
.run()
for col, profile in result.profiles.items():
print(profile)
Error:
KeyError: 'StringColumnProfile'
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
File :5
1 from pydeequ.profiles import *
3 result = ColumnProfilerRunner(spark) \
4 .onData(df) \
----> 5 .run()
7 for col, profile in result.profiles.items():
8 print(profile)
Data Types is mostly string and decimal
Beitragsleitfaden
Rechercherichtung
Start by reproducing the issue in Azure Databricks using the shown pydeequ.profiles import and ColumnProfilerRunner(spark) chain, with mostly string and decimal columns. Trace where result.profiles is built and where the StringColumnProfile key is resolved. Done means the profiler completes without the KeyError and returns profiles for the input columns.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- azure, python, spark
- Bereich
- cloud, data-engineering
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 35/100