classroom-resources / classroom-resources/autograding-python-grader

Autograder does not work with parameterized tests.

Open
#3 5 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11
Forks
14
PR merge metrics
No merged PRs in 30d

Description

The autograder will produce key errors when running test files that have parameterized tests in them.

Here's an example of a parameterized test.

```
@pytest.mark.parametrize("guess, answer, expected",
[(5, 1, 'Lower'), # Lower
(6, 10, 'Higher')]) # Higher
def test_hint_higher_lower(guess, answer, expected):
"""Tests hint_higher_lower."""
# GIVEN: The guess is 5 and the answer is 1
# WHEN: hint_higher_lower(5,1) is run
# THEN: "Lower" is returned
# repeat for Higher.
result = higherlower.hint_higher_lower(guess, answer)
assert expected == result, (f"\nThe Test Failed."
f"\nHere's What was Expected:\n{expected}"
f"\nHere's What Your Function Returned:\n{result}")
```

I've attached a sample error and log archive below.

```
Run classroom-resources/autograding-python-grader@v1
/usr/bin/docker run --name cc4[6](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:7)2fe147022c32b94e38b0f6ce90d4e42922_c34c56 --label cc462f --workdir /github/workspace --rm -e "INPUT_TIMEOUT" -e "INPUT_MAX-SCORE" -e "INPUT_SETUP-COMMAND" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true --entrypoint "/opt/test-runner/bin/run.sh" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/higher-lower-Manurox/higher-lower-Manurox":"/github/workspace" cc462f:e147022c32b94e38b0f6ce90d4e42922 "--timeout=10" "--max-score=1" "--setup-command=sudo -H pip3 install pytest"
/opt/test-runner/bin/run.sh: 1: eval: sudo: not found
TIMEOUT is 600 seconds
MAX_SCORE is 1
Running setup command: sudo -H pip3 install pytest
============================= test session starts ==============================
platform linux -- Python 3.11.2, pytest-[7](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:8).2.2, pluggy-1.5.0
rootdir: /github/workspace, configfile: pyproject.toml
plugins: subtests-0.11.0
5
collected 11 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 1[8](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:9)2, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 334, in pytest_collection
INTERNALERROR> session.perform_collect()
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 667, in perform_collect
INTERNALERROR> hook.pytest_collection_modifyitems(
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/opt/test-runner/runner/__init__.py", line 50, in pytest_collection_modifyitems
INTERNALERROR> items.sort(key=_sort_by_lineno)
INTERNALERROR> File "/opt/test-runner/runner/__init__.py", line 48, in _sort_by_lineno
INTERNALERROR> return TestOrder.lineno(test_id, source)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/opt/test-runner/runner/sort.py", line [9](https://github.com/utd-angm3311-24f/higher-lower-Manurox/actions/runs/10788899832/job/29920670634#step:4:10)4, in lineno
INTERNALERROR> return cls._cache[test_id].lineno
INTERNALERROR> ~~~~~~~~~~^^^^^^^^^
INTERNALERROR> KeyError: 'tests/test_higherlower.py::test_hint_higher_lower[5-1-Lower]'

============================ no tests ran in 0.02s =============================
```

[logs_28192357931.zip](https://github.com/user-attachments/files/16985771/logs_28192357931.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.