bigscience-workshop / bigscience-workshop/biomedical
bc5cdr drug/chemical counts dont match paper
- Dominant language
- Python
- Stars
- 505
- Forks
- 117
- PR merge metrics
- No merged PRs in 30d
Description
When I count the number of chemical and disease entities in the different splits of bc5cdr, I get different numbers than what is reported in the paper and in BLURB
from BLURB
```
Dataset | Task| Train| Dev | Test | Evaluation Metrics
BC5-chem | NER | 5203 | 5347 | 5385 | F1 entity-level
BC5-disease | NER | 4182 | 4244 | 4424 | F1 entity-level
```
vs from bigbio dataset
```python
def get_entity_type_counts(ds):
counter = Counter()
for sample in ds:
for entity in sample['entities']:
counter[entity['type']] += 1
return counter
ds = load_dataset('path/to/bc5cdr', name='bc5cdr_bigbio_kb')
for split in ['train', 'validation', 'test']:
print(get_entity_type_counts(ds[split]))
```
```
# -- train: Counter({'Chemical': 5207, 'Disease': 4363})
# -- val: Counter({'Disease': 4421, 'Chemical': 5352})
# -- test: Counter({'Chemical': 5394, 'Disease': 4534})
```
some of these are approximately 5% off in terms of counts
Contributor guide
Assessment
This issue has not been assessed yet.