`ref_sol` autograder pattern isn't inherently safe
- Dominant language
- Python
- Stars
- 435
- Forks
- 124
- Avg merge
- 5h 40m
- Merged PRs (30d)
- 14
Description
A commonly-used pattern in autograders is the following:
```
def ref_sol():
###CORRECT_CODE###
return x
ref_x = ref_sol()
feedback.check_numpy_array_allclose("x", ref_x, x)
```
A trivial mistake (an extra `#` in `###CORRECT_CODE###`) caused the correct code to **not** be inserted in a situation that recently came up. As a result, `ref_sol` just returned the data from the surrounding scope, i.e. the solution provided by the user. In the example above, this leads to any result `x` being accepted, as long as it is a numpy array.
We should mitigate this problem on the Relate end. My favorite thought for now is that the `ref_sol` gymnastics could be automated, running a function containing the correct code in a clean scope, avoiding leakage and repeated manual implementation of the pattern.
x-ref: https://gitlab.tiker.net/teaching/cs450-f21/-/merge_requests/47#note_53515 (private access only, sorry)
Contributor guide
No contributing guide indexed for this repository
Research direction
Search the repository for ref_sol and ###CORRECT_CODE###, then trace how autograder reference solutions are assembled and evaluated. Reproduce the missing-insertion case and verify that reference code runs in a clean scope without falling back to the learner's surrounding variables; the feedback.check_numpy_array_allclose example defines the reported failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100