allenai / allenai/ScienceWorld

Disallow new steps/actions if the isCompleted flag is set

Abierto
#31 9 comentarios 0 reacciones 0 asignados Ver en GitHub
To be confirmed
Lenguaje dominante
Scala
Estrellas
391
Forks
42
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Certain conditions set the isCompleted flag to be set -- for example, a negative score in the task, signifying task failure:

https://github.com/allenai/ScienceWorld/blob/10dd21a483aca06bbd591a5465353f719dfaffc3/scienceworld/scienceworld.py#L328

Right now we set the isCompleted flag to be true, but if an agent doesn't check for this, it's still possible for it to continue to send commands to the environment. There is a report that this might let the tasks be gamed (especially the forced-choice tasks), as the agent might be able to take steps that ultimately further increase the score. We should likely modify the step() code so that if the isCompleted flag is set, it disallows further steps to be processed, to prevent any issues with agents reporting erroneously high scores in the future.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.