aboutcode-org / aboutcode-org/scancode-toolkit

[GSoC] Train DeBERTa NER model for required phrase prediction

Abierto
#5,137 0 comentarios 0 reacciones 0 asignados Ver en GitHub
GSoC
Lenguaje dominante
Python
Estrellas
2.6k
Forks
791
Merge medio
1 d 12 h
PR fusionados (30 d)
5

Descripción

continuation of #5077. uses the BIOES dataset produced there

## Goal
Finetune DeBERTa-v3-large to predict required phrase spans in license rule text using BIOES labels

## Tasks
- Subword tokenization with label alignment (-100 for continuation tokens)
- Training with class-weighted loss (handle O vs B/I/E/S imbalance)
- Include negative samples (rules without markers, all-O labels) for balanced training
- Evaluation: token F1, exact span match
- ONNX export for CPU inference

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.