ThinkR-open / ThinkR-open/datadiff

[bug] Colonnes factor : classées « character » par detect_column_types mais jamais normalisées par preprocess_dataframe

Open
#28 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
6
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Contexte

Trou silencieux entre deux modules :

  • detect_column_types() (R/data_types.R) classe les factors comme character (intentionnel, testé : test-data-types.R:27-30) → ils reçoivent les règles by_type: character (case_insensitive, trim) ;
  • mais normalize_text() ne transforme que les vecteurs is.character() (R/preprocessing.R) et preprocess_dataframe garde le même prédicat → une colonne factor n'est jamais trimée/lowercasée, sans aucun avertissement.

Résultat : mêmes données, verdict différent selon que la colonne est character ou factor — surprenant pour l'utilisateur (stringsAsFactors, imports SAS/SPSS via haven, etc.).

Reprex

library(datadiff)
ref  <- data.frame(id = 1:2, s = c("alpha", "beta"))
cand_chr <- data.frame(id = 1:2, s = c("ALPHA", " beta"))
cand_fct <- data.frame(id = 1:2, s = factor(c("ALPHA", " beta")))

path <- tempfile(fileext = ".yaml")
write_rules_template(ref, key = "id", path = path,
                     character_case_insensitive = TRUE, character_trim = TRUE)

compare_datasets_from_yaml(ref, cand_chr, key = "id", path = path)$all_passed  # TRUE
compare_datasets_from_yaml(ref, cand_fct, key = "id", path = path)$all_passed  # FALSE (ou type_mismatch)
# même contenu logique, verdicts différents, aucun warning expliquant pourquoi

Critères de succès

  • Comportement décidé et documenté : soit conversion factor → character en préprocessing (avec note dans la doc), soit warning explicite (« column 's' is a factor; text normalization rules are not applied »).
  • Le reprex donne le même verdict pour character et factor (ou un warning actionnable dans le cas factor).
  • Tests couvrant factor côté référence, côté candidat, et des deux côtés.
  • Clarifier au passage le sort des autres types « fourre-tout character » de detect_column_types() (list-columns, integer64) : verdict défini ou warning.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading R/data_types.R and R/preprocessing.R, including detect_column_types(), normalize_text(), and preprocess_dataframe(). Run test-data-types.R and trace the reprex to decide whether factors are converted or warned about. Add coverage for factor columns on the reference, candidate, and both sides; done means matching verdicts or an actionable warning, with list-columns and integer64 behavior clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.