AlibabaResearch / AlibabaResearch/DAMO-ConvAI

Questions for reproducing/ comparing with SpokenWOZ baselines.

Offen
#122 22 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
spokenwoz
Vorherrschende Sprache
Python
Sterne
1.6k
Forks
250
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Hello,

I am currently trying to evaluate models that I trained on SpokenWOZ in order to compare to the baselines you reported in the paper.
Doing this, I'm currently running into some issues:

1. Which evaluation script should be used to report the results? I'm currently using [this](https://github.com/AlibabaResearch/DAMO-ConvAI/blob/3c1890d16b775a449ce77736c146adde6dafd45c/spokenwoz/Finetuning/space_baseline/space_word/space/utils/eval.py) script from space-word and I'm failing to get numbers close to the ones you report (20% less than what you report for inform and success, while reaching higher BLEU score). Also, which settings do you use exactly for the final evaluation (e.g. how do you set `same_eval_as_cambridge` and `use_true_domain_for_ctr_eval`)?
2. Do you have the outputs or the trained model parameters of any of the baseline models available somewhere to verify the evaluation procedure (I need to adapt it to fit to my code base and want to check that I get the same results as you)?
3. When trying to run the training of `space-word` myself, the training runs for a few iterations and then crashes because the `npy`-file `SNG1724` is missing. I see that the dialog exists in the original data, but it is not preprocessed correctly for some reason. Do you have an explanation for this?

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.