allenai / allenai/ScienceWorld

Disallow new steps/actions if the isCompleted flag is set

Ouverte
#31 9 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
To be confirmed
Langage dominant
Scala
Étoiles
391
Forks
42
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

Certain conditions set the isCompleted flag to be set -- for example, a negative score in the task, signifying task failure:

https://github.com/allenai/ScienceWorld/blob/10dd21a483aca06bbd591a5465353f719dfaffc3/scienceworld/scienceworld.py#L328

Right now we set the isCompleted flag to be true, but if an agent doesn't check for this, it's still possible for it to continue to send commands to the environment. There is a report that this might let the tasks be gamed (especially the forced-choice tasks), as the agent might be able to take steps that ultimately further increase the score. We should likely modify the step() code so that if the isCompleted flag is set, it disallows further steps to be processed, to prevent any issues with agents reporting erroneously high scores in the future.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.