LLM grader transcript includes internal base64 prompt-delivery command, triggers content filters
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 711
- Forks
- 46
- PR merge metrics
- No merged PRs in 30d
Description
**Problem**
The LLM rubric grader sends the full session log to the grading model, including skillgrade's internal prompt-delivery command:
echo 'Rmlyc3QgcnVuOi......' | base64 -d > /tmp/.prompt.md
This base64 blob is how skillgrade passes the instruction to the agent CLI — it's an implementation detail, not part of the agent's behavior. However, it appears in the transcript that gets forwarded to the LLM grader.
When using Anthropic-compatible APIs with safety classifiers, the presence of a long base64-encoded string in a user message deterministically triggers content filtering (response comes back as {"content": [], "stop_reason": "end_turn"}). The classifier potentially interprets it as obfuscated payload delivery.
**Reproduction**
Any eval with an instruction longer than ~50 characters will produce a base64 blob large enough to trigger filtering. The grading call fails silently with score 0.
**Suggested fix**
Filter the prompt-encoding command from the session log before building the LLM grader transcript. The line matching echo '...' | base64 -d > /tmp/.prompt.md is internal plumbing — the grader already receives the instruction in the ## Task Instruction section, so the base64 command is redundant information.
Alternatively, expose a config option like redact_internal_commands: true to strip skillgrade's own setup commands from the transcript sent to the LLM grader.
**Environment**
- skillgrade 0.2.1
- agent: claude
- provider: local
- grader_provider: anthropic (via custom gateway)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing where the session log is assembled and where the LLM grader transcript is built; the issue does not name specific files or tests. Reproduce with an instruction longer than 50 characters and an Anthropic-compatible grader, then verify that the internal base64 prompt-delivery command is absent while the task instruction remains and grading no longer silently returns a zero score.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100