awslabs / awslabs/python-deequ

Change message for isUnique method

Abierto
#153 3 comentarios 0 reacciones 0 asignados Ver en GitHub
dependencies enhancement feature request
Lenguaje dominante
Jupyter Notebook
Estrellas
826
Forks
158
Merge medio
9 d 22 h
PR fusionados (30 d)
3

Descripción

For the past day I have been using the Check methods and I have found that isUnique method when using strings is not clear when returning the message of error.

```python
from pydeequ.checks import *
from pydeequ.verification import *
import pydeequ
from pyspark.sql.types import DateType, FloatType, StringType, StructField, StructType, BooleanType
import datetime
from pyspark.sql import SparkSession, Row
from pyspark.sql import DataFrame as SparkDataFrame
from typing import Dict, List
import time

mock_orders =[
{
"date": datetime.date(2019, 12, 28),
"country_code": "FR",
"concept_id": "c73bcdcc-2669-4bf6-81d3-e4ae73fb11fd",
"id": "bar",
"gtv": 27.0,
},
{
"date": datetime.date(2019, 12, 20),
"country_code": "UK",
"concept_id": "123e4567-e89b-12d3-a456-426655440000",
"id": "bar",
"gtv": 27.0,
},
]

orders_reference_mock = spark.createDataFrame(data = mock_orders)

check = Check(spark, CheckLevel.Warning, "Review Check")

checkResult = (VerificationSuite(spark)
.onData(orders_reference_mock)
.addCheck(
check
.isUnique("gtv")
.isUnique("id")
)
.run())

checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, checkResult)

checkResult_df.collect()
```
The results is:
```shell
[Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='UniquenessConstraint(Uniqueness(List(gtv),None))', constraint_status='Failure', constraint_message='Value: 0.0 does not meet the constraint requirement!'),
Row(check='Review Check', check_level='Warning', check_status='Warning', constraint='UniquenessConstraint(Uniqueness(List(id),None))', constraint_status='Failure', constraint_message='Value: 0.0 does not meet the constraint requirement!')]
```

The `constraint_message` is not clear and doesn't give any information. It happens in both cases if it is string or integer.

Is it possible to have a more clear message please? I am putting this as a feature instead of a bug

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza con el punto de entrada Check.isUnique y sigue cómo VerificationSuite.run produce resultados para VerificationResult.checkResultsAsDataFrame. Reproduce los ejemplos de gtv e id y, después, identifica dónde se forma constraint_message. Se considera terminado cuando los resultados de unicidad fallidos explican qué valor o columna causó el fallo, en lugar de informar únicamente 0.0.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
data-engineering
Tipo de issue
Nueva funcionalidad
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
52/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.