Feedback's check_numpy_array_allclose Parameters Not Working

The `report_failure` parameter to `feedback` does not work as intended. Instead of just determining whether to report a failure, it also aborts on failure if set to `True`. Aborting on failure is supposed the be the purpose of the `accuracy_critical` parameter, I think.

See email chain below for background.

From: Andreas Kloeckner
Sent: Saturday, September 10, 2016 7:50PM
To: Bowman, Nathaniel Lee
Cc: Relate-uiuc

Subject: Re: Possible Grading Code Bug

"Bowman, Nathaniel Lee" nlbowma2@illinois.edu writes:

> Hey Andreas,
>
> I had a question (and possible bug) about some grading code for CS 450.
> You're right--at the very least the documentation disagrees with what
> the code actually does. This is where that code lives:

https://github.com/inducer/relate/blob/master/course/page/code_feedback.py

Right now, there are three flags whose meaning is sort of tangled:

report_success
report_failure
accuracy_critical

We should untangle that ASAP to avoid more grading code being written
against the faulty interface. I figure this here would be a reasonable
replacement, where for each thing (numpy array dtype/shape, values)
being checked, you specify whether you want to report success, report
failure, and whether you want to abort when problems happen, e.g. using
the following keyword args:

check_features="rep_ok,rep_failure,abort" (any combination)
check_accuracy="rep_ok,rep_failure,abort" (any combination)

Each would default to "rep_failure,abort". The existing interface would
need to keep working unchanged (although be undocumented at that point).

I'd love to have a patch that does this. :) Could you also stick this
into a bug on github, so it doesn't get forgotten (including my reply)?

Thanks,
Andreas

> This is the grading code I am using:
>
> test_code: |
> import numpy as np
>
> ```
> def compute_soln():
> ###CORRECT_CODE###
> return(P, L, U, sol1, sol2, sol3)
>
> myP, myL, myU, mySol1, mySol2, mySol3 = compute_soln()
>
> feedback.check_numpy_array_allclose('P', myP, P)
> feedback.check_numpy_array_allclose('L', myL, L)
> feedback.check_numpy_array_allclose('U', myU, U)
>
> score = 0.4
>
> if feedback.check_numpy_array_allclose('sol1', mySol1, sol1,
> report_failure=False):
> score += 0.2
> if feedback.check_numpy_array_allclose('sol2', mySol2, sol2,
> report_failure=False):
> score += 0.2
> if feedback.check_numpy_array_allclose('sol3', mySol3, sol3,
> report_failure=False):
> score += 0.2
>
> feedback.set_points(score)
> ```
>
> My hope was that anyone who gets P, L, and U correct will get at least 40%, and then 20% more for each correct sol. What happens is:
> 1) P, L, U correct, nothing else submitted: 40%
> 2) P, L, U correct, sol1 correct, nothing else submitted: 60%
> 3) Same as 2, but sol2 also correct: 80%
> 4) Everything correct: 100%
>
> This is all fine. However,
> 5) P, L, U correct, sol1 submitted and wrong: 0%
> 6) P, L, U correct, sol1 correct, sol2 submitted and wrong: 0%
> 7) Similar with sol3.
> This is puzzling me. Am I doing something wrong with the grading code? Or is this not the expected behavior?
>
> You can test it out yourself if you have access to our repo. It is hw02 problem 1.
>
> Thanks,
> Nate

Start in course/page/code_feedback.py, where check_numpy_array_allclose and the report_failure and accuracy_critical parameters are implemented. Trace how these flags affect reporting and aborting for the grading example in the issue. Done means the existing interface remains compatible while reporting and abort behavior are no longer unintentionally coupled.

Written by the indexing model from the issue text.