huggingface / huggingface/agents-course
[HANDS-ON BUG] Several issues with hands-on quiz
- Dominant language
- MDX
- Stars
- 32.6k
- Forks
- 2.3k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Several issues both with the way the hands-on in Unit 2 is done, both with questions and the evaluation of responses.
**To Reproduce**
Question 1
- using model = HfApiModel() without the model name is considered incorrect. The test description does not specify that a specific model has to be used and the course states that there is a default. The error, "❌ The student has not properly configured the HfApiModel. The model ID is missing, which is necessary for the model to function correctly." / "❌ The student has not specified the model ID for the HfApiModel. The correct model ID should be 'Qwen/Qwen2.5-Coder-32B-Instruct' as shown in the reference solution." is in my view incorrect.
Question 2
- I used `tools=[DuckDuckGoSearchTool(), VisitWebpageTool()],`, the refenrece solution is `tools=[DuckDuckGoSearchTool(), visit_webpage],`. The error I get is "❌ The student used 'VisitWebpageTool' instead of 'visit_webpage' as specified in the reference solution. The tool names should match exactly.". Considering visit_webpage is actually not used in the question, this makes zero sense.
- I used `additional_authorized_imports=["pandas"]` instead of the reference ` additional_authorized_imports=["time", "numpy", "pandas"]` and got "❌ The student only authorized 'pandas' import, while the reference solution authorized 'time', 'numpy', and 'pandas'. All required imports should be included." . Considering the question doesn't actually specific which imports are needed, this doesn't seem to be correct.
Question 3
- Not sure about this one, but I used `agent = CodeAgent(tools=[], model=model, name="some agent", executor_type="e2b")` as documented here as a simplest approach: https://huggingface.co/docs/smolagents/en/tutorials/secure_code_execution , and get the error: "❌ The student's solution uses 'executor_type="e2b"' instead of explicitly creating an instance of 'E2BSandbox()'. This does not meet the requirement of properly configuring the E2B sandbox as specified in the reference solution.". Not sure about this one, but seems odd that what the doc page says is not accepted here. Also the question is very vague.
- Same issue as in Q2 re: Imports.
Question 4
- I used ` tools=[DuckDuckGoSearchTool()], ` instead of ` tools=[custom_tool],` and got: "❌ The student used DuckDuckGoSearchTool, which is not the same as the custom_tool used in the reference solution. While this is not necessarily incorrect, it does not meet the specific requirement of the challenge.". This doesn't seem to make sense, considering the question doesn't specifiy anything re: tool to use or not.
Question 5
- One of the assessment criteria is "Alternative model option is provided", which I didn't provide, and the error states "❌ The student's solution only includes the HfApiModel and does not provide an alternative model option using LiteLLMModel as specified in the reference solution.". Again IMO this error doesn't make sense, considering the assessment criteria is vague and doesn't specify what's needed. I suspect if I declare a second instance of HfApiModel() with another model I'd get the same error.
All in all, this code challenge frustrated me for the lack of clarity in the questions and strictness in the answer verification, and it'd benefit from a e2e revision. In a way, if an LLM is used to evaluate, it has the effect of showing they shouldn't be used to do this in real applications :-( .
**Screenshots**
Information provided above
**Additional context**
--
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.