bigscience-workshop / bigscience-workshop/data_tooling

Reason for not applying remove_non_prining_characters normalization

Ouverte
#416 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
HTML
Étoiles
91
Forks
47
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

Hi,

We are much inspired by this great work and are in the process of cleaning our data. However, if we understand correctly, the `remove_non_prining_characters` normalization step is not used for the final cleaning. Do you have any thoughts on why this should not be used?

https://github.com/bigscience-workshop/data_tooling/blob/e28064ec7fb38af5143cafc896e9423a8b12392d/ac_dc/normalization.py#L5

There you have this:
```
non_printing_characters_re = re.compile(
f"[{''.join(map(chr, list(range(0,32)) + list(range(127,160))))}]"
)
```

Which we modified, to keep newlines (`\n`) and tabs (`\t`), and to also remove soft-hyphens, non-breaking spaces, and zero-width space:

```
additional_chars_to_remove = [160, 173, 8203]
non_printing_characters_re = re.compile(
f"[{''.join(map(chr, list(range(0,9)) + list(range(11, 32)) + list(range(127,160)) + additional_chars_to_remove))}]"
)
```

There could of course be more characters that one may want to remove.

To be clear, I am writing this here for two reasons:
1. To get your feedback. Do you think this is a good idea to use for the final data cleaning?
2. If so, this could be incorporated into this repository to help other people that might be thinking about this.

Thanks for your amazing contributions!

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Read ac_dc/normalization.py at the referenced non-printing-character regex, then trace the final cleaning path to confirm whether this normalization is applied. Compare the proposed character ranges and retained newlines and tabs with the repository’s current behavior. Done means reaching a maintainer-backed decision and, if accepted, incorporating the guidance for future data cleaning.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
data
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.