microsoft / microsoft/Test_Awareness_Steering
Model awareness vs evaluation awareness?
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 14
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
Hey
In your judge prompt (https://github.com/microsoft/Test_Awareness_Steering/blob/main/prompts/judgePromptEvidenceBoth.txt) you first describe "Evaluation Awareness" but later require a JSON key "model_awareness". Later you talk more about "evaluation awareness" and only in the few last lines you mention "model_awareness".
This might confuse the judge? At least, it confused me : )
So, I think my main question is: was this on purpose, or just an accidental mismatch? I'm working on some follow-up and if this is an accident then I'm gonna fix that in my code.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with prompts/judgePromptEvidenceBoth.txt and compare the uses of “Evaluation Awareness” and “model_awareness.” Confirm which terminology the judge prompt intends to use, then make the terminology consistent or document the distinction. Done means the prompt’s concept and JSON key are unambiguous.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100