awslabs / awslabs/python-deequ
ConstraintSuggestionRunner generates incorrect code_for_constraint when ")" met in dataframe values
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- Jupyter Notebook
- Sterne
- 826
- Forks
- 158
- Ø Merge
- 9 T. 22 Std.
- Gemergte PRs (30 T.)
- 3
Beschreibung
**Describe the bug**
Function: **__s2p_filter** in Class ConstraintSuggestionRunner causes incorrect code_for_constraint in produced constraints suggestions.
@staticmethod
def __s2p_filter(code: str):
"""
Scala -> Python translator for the constraint suggestions code
A method that returns the python translation of the scala constraint suggestion
:param str code: a scala constraint suggestion
:return code that is translated to look more like python code
"""
if ' _ ' in code:
code = code.replace(' _ ', ' lambda x: x ')
if 'Some(' in code:
# Usually at the end as 'where' or 'hint' strings as optional
code = code.replace('Some(', '')[:-1]
if 'Array(' in code:
# TODO: what if multiple?
# TODO: Probz redo with regex
start = code.index('Array(') + len('Array(')
for idx in range(start, len(code)):
if code[idx] == ')':
code = code[:idx] + ']' + code[idx + 1:]
code = code.replace('Array(', '[')
break
if 'Seq(' in code:
# TODO: what if multiple?
# TODO: Probz redo with regex
start = code.index('Seq(') + len('Seq(')
for idx in range(start, len(code)):
if code[idx] == ')':
code = code[:idx] + ']' + code[idx + 1:]
code = code.replace('Seq(', '[')
break
return code
Due to searching in the string for the first occurrence of ")", as you can see in the above-inserted method's code, it can generate constraint suggestions incorrectly.
For DataFrame which contains example values in its column: "some string input (with additional explanation here)" it can replace with the square right bracket found within column value and produce such code_for_constraint:
**.isContainedIn("d", ["some string input (with additional explanation here]"))** - which isn't valid python code
**To Reproduce**
Here's adjusted code from the tutorial to reflect incorrect behavior:
`
from pydeequ.suggestions import *
from pyspark.sql import SparkSession, Row
df = (spark.sparkContext.parallelize([
Row(d="some string input (with additional explanation here)"),
Row(d="some string input (with additional explanation here)"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input")
]).toDF())
suggestionResult = ConstraintSuggestionRunner(spark) \
.onData(df) \
.addConstraintRule(DEFAULT()) \
.run()
for code in suggestionResult["constraint_suggestions"]:
print(f"column_name: {code['column_name']}")
print(f"code_for_constraint: {code['code_for_constraint']}", "\n")
`
**Expected behavior**
I expect adjusting **__s2p_filter** method to replace for closing square bracket correct closing bracket from Array or Seq Scala's code instead of the first found.
Expected value from above **To Reproduce** section:
**.isContainedIn("d", ["some string input", "some string input (with additional explanation here)"])**
What we get currently from **To Reproduce** section:
**.isContainedIn("d", ["some string input", "some string input (with additional explanation here]"))**
**Screenshots**


Beitragsleitfaden
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Beginnen Sie bei ConstraintSuggestionRunner.__s2p_filter, dem im Issue gezeigten Scala-to-Python-Übersetzer, und reproduzieren Sie das Problem mit dem bereitgestellten Spark DataFrame-Beispiel. Überprüfen Sie, dass Klammern innerhalb von DataFrame-Werten nicht die schließende Klammer für die Konvertierung von Array oder Seq bestimmen; abgeschlossen ist dies, wenn der generierte code_for_constraint dem erwarteten isContainedIn-Ausdruck entspricht und gültiges Python bleibt.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python, scala, spark
- Bereich
- data-engineering
- Issue-Typ
- Bug
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Klar beschrieben
- Anfängerfreundlichkeit
- 70/100