inducer / inducer/relate

Autograder holes

Open
#295 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
435
Forks
124
Avg merge
5h 40m
Merged PRs (30d)
14

Description

It is easy to trick the autograder into accepting an object that cheats the correctness checks. Here's two examples that bypass the [scalar value check](https://github.com/inducer/relate/blob/master/course/page/code_feedback.py#L135), tested on CS357 quiz 17:

```
# Type not being checked when is_number attribute is True
# https://github.com/inducer/relate/blob/master/course/page/code_feedback.py#L144
class Evil:
def __init__(self):
self.is_number = True
def __abs__(self):
return self
def __rsub__(self, o):
return self
def __lt__(self, o):
return True

answer = Evil()
```

```
# Same idea, since we subclassed one of the accepted types
# isinstance() will let us through. Might need to change the
# superclass based on what the accepted answer type is, since
# __rsub__ doesn't get called if they're the same type.
class Evil2(np.float64):
def __abs__(self):
return self
def __rsub__(self, o):
return self
def __lt__(self, o):
return True

answer = Evil2()
```
![2017-03-13 01_41_16](https://cloud.githubusercontent.com/assets/14020276/23844121/37e0fb9a-078e-11e7-85f9-a56f895c2838.png)

Changing `isinstance(x, (...))` to `type(x) in [...]` (and fixing the `is_number` check) would prevent this particular trick by disallowing subclasses, though I'm not sure if that would be secure either.

Sorry if you don't want this code floating around in public, let me know and I'll remove it ASAP.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the scalar value check in course/page/code_feedback.py around lines 135 and 144, then reproduce the two CS357 quiz 17 examples. Evaluate validation that rejects both the is_number and subclass bypasses; done means neither example can pass the correctness check, with regression coverage added where the grading tests enter this path.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.