allenai / allenai/RL4LMs

Question about the classifier used for IntentAccuracyDailyDialog.

Open
#71 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
201
PR merge metrics
No merged PRs in 30d

Description

According to the source code of [class IntentAccuracyDailyDialog(BaseMetric)](https://github.com/allenai/RL4LMs/blob/97df0bd2f7406a906206c9610aea795fbf52884c/rl4lms/envs/text_generation/metric.py#L663), the intent likelihood of utterances on DailyDialog is computed by `rajkumarrrk/roberta-daily-dialog-intent-classifier`.

However, according to the `config.json` of this classifier, it is used for emotion classification, with four labels: joy, optimism, anger, and sadness, while the intent labels on [DailyDialog](https://arxiv.org/pdf/1710.03957.pdf) should be Inform, Questions, Directives, and Commissive instead.

So my question is: **Is this classifier already fine-tuned on intent classification of DailyDialog utterances?**

Empirically, i obeserve that the classification results of ground truth utterances in DailyDialog by this classifier are unbalanced and not well-aligned to the labelled intent distribution, as shown below.

+ classification results on test set

| |label-0|label-1|label-2|label-3|Intent Accuracy|
|---------|-------|--------|--------|-------|------------------------------|
|classification on ground truth|0.7102|0.0055|0.0275|0.2071|0.6147|
|intent labels in DailyDialog|0.4988|0.2231|0.1565|0.1213|-|
|classification on SFT generation|0.5363|0.1591|0.0944|0.2100|0.4034|

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.