allenai / allenai/pawls

BUG: duplicated tokens should not be allowed in pdf_structure tokens list

未关闭
#186 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
435
派生
80
PR 合并指标
30 天内没有已合并 PR

描述

In the pdfplumber preprocess pipeline I've found that duplicated tokens may exist. Specifically in the `obtain_word_tokens` of the `pdfplumber.py` file, one should put a `.drop_duplicates` before converting the dataframe to list.

```
word_tokens = df.apply(self.convert_to_pagetoken, axis=1).drop_duplicates(keep="first").tolist()
```
It can happen in some cases that the tokens from a PAWLS pdf structure appear duplicated and this messes up things a bit when indexing from the annotation file.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。