doccano / doccano/doccano-transformer

Bug in Sentence Offset

Open
#24 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
108
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Hello, I want to report a bug. I have a Sentence like this:

`As a administrator, I want to refund sponsorship money that was processed via stripe, so that people get their monies back.`

When I try to convert it to CoNLL the span is not converted well. I then debug the library and found that the offset is wrong. Here is the output of the offset:

```
As 0
a 3
administrator, 3
I 20
want 22
to 25
refund 30
sponsorship 37
money 49
that 55
was 60
processed 64
via 74
stripe, 78
so 86
that 89
people 94
get 101
their 103
monies 111
back. 118
```

As you can see, in the second and third lines, the offset is the same (3 and 3, while It should be 3 and 5). This behavior makes the span undetected in the conversion process.

It seems that the `get_offsets` function in `utils.py` checks the equality in the sequence of characters to decide about the offsets.

```python
def get_offsets(
text: str,
tokens: List[str],
start: Optional[int] = 0) -> List[int]:
"""Calculate char offsets of each tokens.

Args:
text (str): The string before tokenized.
tokens (List[str]): The list of the string. Each string corresponds
token.
start (Optional[int]): The start position.
Returns:
(List[str]): The list of the offset.
"""
offsets = []
i = 0
for token in tokens:
for j, char in enumerate(token):
while char != text[i]:
i += 1

if j == 0:
offsets.append(i + start)
return offsets
```

It will be a problem if the last character of the previous word is the same as the first character of the next word. I'm still looking for a fix to this problem.

Cheers

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.