BUG: duplicated tokens should not be allowed in pdf_structure tokens list
Ouverte
- Langage dominant
- Python
- Étoiles
- 435
- Forks
- 80
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
In the pdfplumber preprocess pipeline I've found that duplicated tokens may exist. Specifically in the `obtain_word_tokens` of the `pdfplumber.py` file, one should put a `.drop_duplicates` before converting the dataframe to list.
```
word_tokens = df.apply(self.convert_to_pagetoken, axis=1).drop_duplicates(keep="first").tolist()
```
It can happen in some cases that the tokens from a PAWLS pdf structure appear duplicated and this messes up things a bit when indexing from the annotation file.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.