Autograder holes
- Dominant language
- Python
- Stars
- 435
- Forks
- 124
- Avg merge
- 5h 40m
- Merged PRs (30d)
- 14
Description
It is easy to trick the autograder into accepting an object that cheats the correctness checks. Here's two examples that bypass the [scalar value check](https://github.com/inducer/relate/blob/master/course/page/code_feedback.py#L135), tested on CS357 quiz 17:
```
# Type not being checked when is_number attribute is True
# https://github.com/inducer/relate/blob/master/course/page/code_feedback.py#L144
class Evil:
def __init__(self):
self.is_number = True
def __abs__(self):
return self
def __rsub__(self, o):
return self
def __lt__(self, o):
return True
answer = Evil()
```
```
# Same idea, since we subclassed one of the accepted types
# isinstance() will let us through. Might need to change the
# superclass based on what the accepted answer type is, since
# __rsub__ doesn't get called if they're the same type.
class Evil2(np.float64):
def __abs__(self):
return self
def __rsub__(self, o):
return self
def __lt__(self, o):
return True
answer = Evil2()
```

Changing `isinstance(x, (...))` to `type(x) in [...]` (and fixing the `is_number` check) would prevent this particular trick by disallowing subclasses, though I'm not sure if that would be secure either.
Sorry if you don't want this code floating around in public, let me know and I'll remove it ASAP.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the scalar value check in course/page/code_feedback.py around lines 135 and 144, then reproduce the two CS357 quiz 17 examples. Evaluate validation that rejects both the is_number and subclass bypasses; done means neither example can pass the correctness check, with regression coverage added where the grading tests enter this path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100