aboutcode-org / aboutcode-org/scancode-toolkit

[GSoC] Build NER training dataset from annotated rules

Open
#5,077 0 comments 0 reactions 0 assignees View on GitHub
GSoC
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

Tasks:
- Parse .RULE files and extract extract required phrase markers
- Propagate annotations (with CLI commands from #3924 ) to unannotated rules (with matching text) to expand the dataset
- Improve composite rules (AND/OR expressions where some aren't marked)
- Convert to BIOES labels
- Split into train/val/test by rule family
- Validate dataset quality

Timeline: Weeks 1-2 (current)

Deliverable: BIOES dataset ready for training (train/val/test splits)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.