allenai / allenai/ScienceWorld

Disallow new steps/actions if the isCompleted flag is set

Aberta
#31 9 comentários 0 reações 0 responsáveis Ver no GitHub
To be confirmed
Linguagem predominante
Scala
Estrelas
391
Forks
42
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

Certain conditions set the isCompleted flag to be set -- for example, a negative score in the task, signifying task failure:

https://github.com/allenai/ScienceWorld/blob/10dd21a483aca06bbd591a5465353f719dfaffc3/scienceworld/scienceworld.py#L328

Right now we set the isCompleted flag to be true, but if an agent doesn't check for this, it's still possible for it to continue to send commands to the environment. There is a report that this might let the tasks be gamed (especially the forced-choice tasks), as the agent might be able to take steps that ultimately further increase the score. We should likely modify the step() code so that if the isCompleted flag is set, it disallows further steps to be processed, to prevent any issues with agents reporting erroneously high scores in the future.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.