bigscience-workshop / bigscience-workshop/biomedical

bc5cdr drug/chemical counts dont match paper

Open
#597 3 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Python
Stars
505
Forks
117
PR merge metrics
No merged PRs in 30d

Description

When I count the number of chemical and disease entities in the different splits of bc5cdr, I get different numbers than what is reported in the paper and in BLURB

from BLURB
```
Dataset | Task| Train| Dev | Test | Evaluation Metrics

BC5-chem | NER | 5203 | 5347 | 5385 | F1 entity-level
BC5-disease | NER | 4182 | 4244 | 4424 | F1 entity-level
```

vs from bigbio dataset
```python
def get_entity_type_counts(ds):
counter = Counter()
for sample in ds:
for entity in sample['entities']:
counter[entity['type']] += 1
return counter

ds = load_dataset('path/to/bc5cdr', name='bc5cdr_bigbio_kb')
for split in ['train', 'validation', 'test']:
print(get_entity_type_counts(ds[split]))
```

```
# -- train: Counter({'Chemical': 5207, 'Disease': 4363})
# -- val: Counter({'Disease': 4421, 'Chemical': 5352})
# -- test: Counter({'Chemical': 5394, 'Disease': 4534})
```

some of these are approximately 5% off in terms of counts

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.