4paradigm / 4paradigm/AutoX

AutoX_NLP/ nlp_feature.py, OOV问题优化

Open
#51 0 comments 0 reactions 0 assignees View on GitHub
AutoX_NLP call-for-contributions
Dominant language
Jupyter Notebook
Stars
551
Forks
143
PR merge metrics
No merged PRs in 30d

Description

当前Word2Vec和Glove模型无法处理测试数据中未见过的词,需要对测试数据重新进行词表构建,对整体效果影响较大。
代码链接:[https://github.com/4paradigm/AutoX/blob/master/autox/autox_nlp/feature_engineer/nlp_feature.py](url)

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue points to nlp_feature.py in autox/autox_nlp/feature_engineer/. Examine how Word2Vec and GloVe models are built and applied, focusing on vocabulary handling for OOV (out-of-vocabulary) words. Look at the test data pipeline to see where the vocabulary is constructed. The goal is to modify the feature engineering to better handle unseen words, likely by adjusting the vocabulary building or embedding fallback strategy.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.