mgechev / mgechev/skillgrade

LLM grader transcript includes internal base64 prompt-delivery command, triggers content filters

Open
#29 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
711
Forks
46
PR merge metrics
No merged PRs in 30d

Description

**Problem**

The LLM rubric grader sends the full session log to the grading model, including skillgrade's internal prompt-delivery command:

echo 'Rmlyc3QgcnVuOi......' | base64 -d > /tmp/.prompt.md

This base64 blob is how skillgrade passes the instruction to the agent CLI — it's an implementation detail, not part of the agent's behavior. However, it appears in the transcript that gets forwarded to the LLM grader.

When using Anthropic-compatible APIs with safety classifiers, the presence of a long base64-encoded string in a user message deterministically triggers content filtering (response comes back as {"content": [], "stop_reason": "end_turn"}). The classifier potentially interprets it as obfuscated payload delivery.

**Reproduction**

Any eval with an instruction longer than ~50 characters will produce a base64 blob large enough to trigger filtering. The grading call fails silently with score 0.

**Suggested fix**

Filter the prompt-encoding command from the session log before building the LLM grader transcript. The line matching echo '...' | base64 -d > /tmp/.prompt.md is internal plumbing — the grader already receives the instruction in the ## Task Instruction section, so the base64 command is redundant information.

Alternatively, expose a config option like redact_internal_commands: true to strip skillgrade's own setup commands from the transcript sent to the LLM grader.

**Environment**

- skillgrade 0.2.1
- agent: claude
- provider: local
- grader_provider: anthropic (via custom gateway)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing where the session log is assembled and where the LLM grader transcript is built; the issue does not name specific files or tests. Reproduce with an instruction longer than 50 characters and an Anthropic-compatible grader, then verify that the internal base64 prompt-delivery command is absent while the task instruction remains and grading no longer silently returns a zero score.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.