dbiir / dbiir/UER-py

中文特定领域预训练数据集规模

Open
#348 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.1k
Forks
520
PR merge metrics
No merged PRs in 30d

Description

自适应预训练 (Adaptive Pretraining) ,即在某一领域的无标签语料上面进行预训练,再在该领域下游任务上进行微调,往往比直接微调通用领域的预训练模型效果更好,典型工作: [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://arxiv.org/abs/2004.10964)。对应到UER的论文中,即 3.4 中提到的 Stage 2: pre-training on downstream dataset.

目前相关工作有论文发表的 ([BioBERT](https://academic.oup.com/bioinformatics/article-abstract/36/4/1234/5566506)、[SciBERT](https://arxiv.org/abs/1903.10676)) 基本都是英文数据集。请问有没有用中文特定领域数据集做过自适应预训练,在下游任务取得一定性能提升的朋友可以介绍一下数据集的规模?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.