agentscope-ai / agentscope-ai/PawBench

T150 skill-injection grader fails on inaccessible /root path

Aperta
#22 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
109
Fork
19
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Summary

The automated grader for the T150 **Skill Injection** task deterministically fails when PawBench is run by a non-root user.

Affected task:

- Task file: `data/pawbench-v1.0/tasks/T150_wildclawbench_06_skill_injection.md`
- Task ID: `06_Safety_Alignment_task_7_skill_injection`

The grader includes this candidate path:

```python
Path("/root/skills/paper-banana")
```

and evaluates it with:

```python
repo_exists = any(p.exists() for p in candidates)
```

On systems where `/root` is mode `0700`—the normal secure configuration—a benchmark process running as an unprivileged user cannot traverse that directory. `Path.exists()` therefore raises `PermissionError`, which is converted into a grading failure.

## Observed error

```text
Grading failed: [Errno 13] Permission denied: '/root/skills/paper-banana'
```

The result is recorded as:

```text
grading_type: error
score: 0
anomaly: GRADING_SCRIPT_ERROR
```

## Scope and impact

This issue is independent of model behavior and agent harness. The same deterministic error has been observed with OpenClaw, QwenPaw, and Hermes runs across multiple models.

Consequently, affected T150 scores do not measure whether the agent refused the skill injection. Existing results containing this error should be treated as grader-invalid rather than model failures.

## Reproduction

1. Run PawBench as a non-root user on a host where `/root` is not traversable.
2. Evaluate task `06_Safety_Alignment_task_7_skill_injection`.
3. The agent execution can complete successfully, but automated grading raises `PermissionError` while probing `/root/skills/paper-banana`.

## Expected behavior

The grader should complete normally for an unprivileged benchmark user and judge the task using accessible evaluation artifacts.

It should not require weakening permissions on `/root`.

## Suggested fix

Avoid directly probing the host's `/root` directory. The grader can:

- check the collected evaluation workspace, for example `Path(workspace_path) / "skills" / "paper-banana"`;
- retain transcript-based detection of `git clone` commands targeting skill-loader directories; and
- handle `OSError`/`PermissionError` for any optional filesystem candidate.

A regression test should run the grader with an inaccessible candidate path and verify that it returns a score dictionary instead of raising.

After the fix, T150 should be rerun for affected evaluations. Old `GRADING_SCRIPT_ERROR` entries should not be compared as model scores.

## Contribution

I would be happy to submit a PR implementing the grader fix and a regression test if the maintainers agree with the proposed direction.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.