huggingface / huggingface/setfit
How to improve the accuracy while classifying short text with less context
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
Hi, my usecase is to classify Job Title into Functional Areas. I finetuned `all-mpnet-base-v2` with the help of setfit by providing some 10+ examples for each class (Functional Areas).
I got `82%` accuracy on running the evaluation on my test set. I observed some of the simple & straightforward job titles are classified into wrong label with `0.6` score.
For example:
```
Query: SDET
Predicted Label: Big Data / DWH / ETL
Confidence Scores:
Label: Accounting / Finance, Confidence: 0.0111
Label: Backend Development, Confidence: 0.0140
Label: Big Data / DWH / ETL, Confidence: 0.6092
```
Here **SDET** should have labelled as `QA / SDET` but it is classified to `Big Data / DWH / ETL` with `0.62` score. Few shot examples used for both classes doesn't have anything in common which could confuse the model except one example whose title is `Data Quality Engineer` and it is under `Big Data / DWH / ETL`.
**Few shot examples** (added only for 2 here)
```py
{ "QA / SDET": [
"Quality Assurance Engineer",
"Software Development Engineer in Test (SDET)",
"QA Automation Engineer",
"Test Engineer",
"QA Analyst",
"Manual Tester",
"Automation Tester",
"Performance Test Engineer",
"Security Test Engineer",
"Mobile QA Engineer",
"API Tester",
"Load & Stress Test Engineer",
"Senior QA Engineer",
"Test Automation Architect",
"QA Lead",
"QA Manager",
"End-to-End Tester",
"Game QA Tester",
"UI/UX Tester",
"Integration Test Engineer",
"Quality Control Engineer",
"Test Data Engineer",
"DevOps QA Engineer",
"Continuous Integration (CI) Tester",
"Software Test Consultant"
],
"Big Data / DWH / ETL": [
"Big Data Engineer",
"Data Warehouse Developer",
"ETL Developer",
"Hadoop Developer",
"Spark Developer",
"Data Engineer",
"Data Integration Specialist",
"Data Pipeline Engineer",
"Data Architect",
"Database Administrator",
"ETL Architect",
"Data Lake Engineer",
"Informatica Developer",
"DataOps Engineer",
"BI Developer",
"Data Migration Specialist",
"Data Warehouse Architect",
"ETL Tester",
"Big Data Platform Engineer",
"Apache Kafka Engineer",
"Snowflake Developer",
"Data Quality Engineer",
"Data Ingestion Engineer",
"Big Data Consultant",
"ETL Manager"
]
}
```
**TrainingArgs**
```py
args = TrainingArguments(
batch_size=16,
num_epochs=1,
evaluation_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
)
```
**Here is the complete set of functional areas.**
```py
functional_areas = [
"Accounting / Finance",
"Backend Development",
"Big Data / DWH / ETL",
"Brand Management",
"Content Writing",
"Customer Service",
"Data Analysis / Business Intelligence",
"Data Science / Machine Learning",
"Database Admin / Development",
"DevOps / Cloud",
"Embedded / Kernel Development",
"Event Management",
"Frontend Development",
"Full-Stack Development",
"Functional / Technical Consulting",
"General Management / Strategy",
"IT Management / IT Support",
"IT Security",
"Mobile Development",
"Network Administration",
"Online Marketing",
"Operations Management",
"PR / Communications",
"QA / SDET",
"SEO / SEM",
"Sales / Business Development"
]
```
My guess is accuracy is low because of short text (which is just job title). Please suggest few things which I can try out to improve the accuracy of the model.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.