BUG: duplicated tokens should not be allowed in pdf_structure tokens list
未关闭
- 主要语言
- Python
- 星标
- 435
- 派生
- 80
- PR 合并指标
- 30 天内没有已合并 PR
描述
In the pdfplumber preprocess pipeline I've found that duplicated tokens may exist. Specifically in the `obtain_word_tokens` of the `pdfplumber.py` file, one should put a `.drop_duplicates` before converting the dataframe to list.
```
word_tokens = df.apply(self.convert_to_pagetoken, axis=1).drop_duplicates(keep="first").tolist()
```
It can happen in some cases that the tokens from a PAWLS pdf structure appear duplicated and this messes up things a bit when indexing from the annotation file.
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。