allenai / allenai/ScienceWorld

Disallow new steps/actions if the isCompleted flag is set

Aperta
#31 9 commenti 0 reazioni 0 assegnatari Vedi su GitHub
To be confirmed
Lingua principale
Scala
Stelle
391
Fork
42
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Certain conditions set the isCompleted flag to be set -- for example, a negative score in the task, signifying task failure:

https://github.com/allenai/ScienceWorld/blob/10dd21a483aca06bbd591a5465353f719dfaffc3/scienceworld/scienceworld.py#L328

Right now we set the isCompleted flag to be true, but if an agent doesn't check for this, it's still possible for it to continue to send commands to the environment. There is a report that this might let the tasks be gamed (especially the forced-choice tasks), as the agent might be able to take steps that ultimately further increase the score. We should likely modify the step() code so that if the isCompleted flag is set, it disallows further steps to be processed, to prevent any issues with agents reporting erroneously high scores in the future.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.