inducer / inducer/relate

`ref_sol` autograder pattern isn't inherently safe

Open
#828 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
435
Forks
124
Avg merge
5h 40m
Merged PRs (30d)
14

Description

A commonly-used pattern in autograders is the following:
```
def ref_sol():
###CORRECT_CODE###
return x

ref_x = ref_sol()

feedback.check_numpy_array_allclose("x", ref_x, x)
```
A trivial mistake (an extra `#` in `###CORRECT_CODE###`) caused the correct code to **not** be inserted in a situation that recently came up. As a result, `ref_sol` just returned the data from the surrounding scope, i.e. the solution provided by the user. In the example above, this leads to any result `x` being accepted, as long as it is a numpy array.

We should mitigate this problem on the Relate end. My favorite thought for now is that the `ref_sol` gymnastics could be automated, running a function containing the correct code in a clean scope, avoiding leakage and repeated manual implementation of the pattern.

x-ref: https://gitlab.tiker.net/teaching/cs450-f21/-/merge_requests/47#note_53515 (private access only, sorry)

Contributor guide

No contributing guide indexed for this repository

Research direction

Search the repository for ref_sol and ###CORRECT_CODE###, then trace how autograder reference solutions are assembled and evaluated. Reproduce the missing-insertion case and verify that reference code runs in a clean scope without falling back to the learner's surrounding variables; the feedback.check_numpy_array_allclose example defines the reported failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
backend, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.