awslabs / awslabs/python-deequ

ConstraintSuggestionRunner generates incorrect code_for_constraint when ")" met in dataframe values

Aperta
#51 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
bug enhancement
Lingua principale
Jupyter Notebook
Stelle
826
Fork
158
Merge medio
9g 22h
PR unite (30g)
3

Descrizione

**Describe the bug**
Function: **__s2p_filter** in Class ConstraintSuggestionRunner causes incorrect code_for_constraint in produced constraints suggestions.

@staticmethod
def __s2p_filter(code: str):
"""
Scala -> Python translator for the constraint suggestions code
A method that returns the python translation of the scala constraint suggestion

:param str code: a scala constraint suggestion

:return code that is translated to look more like python code
"""
if ' _ ' in code:
code = code.replace(' _ ', ' lambda x: x ')

if 'Some(' in code:
# Usually at the end as 'where' or 'hint' strings as optional
code = code.replace('Some(', '')[:-1]

if 'Array(' in code:
# TODO: what if multiple?
# TODO: Probz redo with regex
start = code.index('Array(') + len('Array(')
for idx in range(start, len(code)):
if code[idx] == ')':
code = code[:idx] + ']' + code[idx + 1:]
code = code.replace('Array(', '[')
break

if 'Seq(' in code:
# TODO: what if multiple?
# TODO: Probz redo with regex
start = code.index('Seq(') + len('Seq(')
for idx in range(start, len(code)):
if code[idx] == ')':
code = code[:idx] + ']' + code[idx + 1:]
code = code.replace('Seq(', '[')
break

return code

Due to searching in the string for the first occurrence of ")", as you can see in the above-inserted method's code, it can generate constraint suggestions incorrectly.

For DataFrame which contains example values in its column: "some string input (with additional explanation here)" it can replace with the square right bracket found within column value and produce such code_for_constraint:
**.isContainedIn("d", ["some string input (with additional explanation here]"))** - which isn't valid python code

**To Reproduce**
Here's adjusted code from the tutorial to reflect incorrect behavior:
`
from pydeequ.suggestions import *
from pyspark.sql import SparkSession, Row

df = (spark.sparkContext.parallelize([
Row(d="some string input (with additional explanation here)"),
Row(d="some string input (with additional explanation here)"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input"),
Row(d="some string input")
]).toDF())

suggestionResult = ConstraintSuggestionRunner(spark) \
.onData(df) \
.addConstraintRule(DEFAULT()) \
.run()

for code in suggestionResult["constraint_suggestions"]:
print(f"column_name: {code['column_name']}")
print(f"code_for_constraint: {code['code_for_constraint']}", "\n")
`

**Expected behavior**
I expect adjusting **__s2p_filter** method to replace for closing square bracket correct closing bracket from Array or Seq Scala's code instead of the first found.

Expected value from above **To Reproduce** section:
**.isContainedIn("d", ["some string input", "some string input (with additional explanation here)"])**
What we get currently from **To Reproduce** section:
**.isContainedIn("d", ["some string input", "some string input (with additional explanation here]"))**

**Screenshots**
![image](https://user-images.githubusercontent.com/49615819/123931681-07d7ca80-d991-11eb-8e58-9aa39647407d.png)
![image](https://user-images.githubusercontent.com/49615819/123931831-2c33a700-d991-11eb-9763-01f5bd3914d2.png)

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da ConstraintSuggestionRunner.__s2p_filter, il traduttore da Scala-to-Python mostrato nell’issue, e riproduci il problema con l’esempio di Spark DataFrame fornito. Verifica che le parentesi all’interno dei valori del dataframe non determinino la parentesi quadra di chiusura per la conversione di Array o Seq; il lavoro è completato quando il code_for_constraint generato corrisponde all’espressione isContainedIn prevista e rimane Python valido.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, scala, spark
Ambito
data-engineering
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
70/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.