OpenHands / OpenHands/extensions

Audit: Skills Frontmatter Quality Issues

Open
#216 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
148
Forks
90
Avg merge
1d 17h
Merged PRs (30d)
36

Description

Skills Frontmatter Audit Report

This issue documents findings from an audit of all 64 SKILL.md files across skills/ and plugins/.


⚠️ CRITICAL CONTEXT: How Triggers Work in the SDK

Triggers use simple case-insensitive substring matching:

# From openhands-sdk/openhands/sdk/skills/skill.py
def match_trigger(self, message: str) -> str | None:
    if isinstance(self.trigger, KeywordTrigger):
        message_lower = message.lower()
        for keyword in self.trigger.keywords:
            if keyword.lower() in message_lower:  # SUBSTRING MATCH!
                return keyword

This means:

  • Trigger "git" will match "github", "digit", "legitimate"
  • Trigger "uv" will match "cuve", "suvey", "manuver"
  • Trigger "azure" will match "lazure" or any text containing "azure"

This makes many of the issues below much more severe than simple ambiguity - they cause actual false positive activations.


🔴 CRITICAL ISSUES

1. Skills Without Triggers (4 skills)

These skills cannot be discovered through keyword matching:

Skill Impact
skills/add-skill Users cannot naturally discover how to import skills
skills/frontend-design Will not activate when users ask about building UIs
skills/skill-creator Will not activate when users want to create skills
skills/theme-factory Will not activate when users want to style artifacts

Recommendation: Add appropriate keyword triggers like triggers: [frontend design, ui design, web interface] for frontend-design, etc.

2. Dangerously Generic Triggers (CAUSES FALSE POSITIVES)

Due to substring matching, these triggers will activate in unintended contexts:

Trigger Skills Using It False Positive Examples
git bitbucket, github, gitlab, vulnerability-remediation/github "digital", "legitimate", "github" (activates bitbucket skill!)
uv skills/uv "cuve", "suvey", "manuver"
azure azure-devops "lazure", any Azure mention
ticket linear "ticketing", "sticket"
ssh skills/ssh "pussh", "crassh"
npm skills/npm (less common but possible)
k8s skills/kubernetes (reasonable, well-known abbreviation)

Recommendation:

  • Change git to more specific triggers: github repository, gitlab repo, bitbucket repo
  • Change uv to uv python, uv package, uv.lock
  • Change azure to azure devops, azure repos
  • Change ticket to linear ticket, linear issue
  • Change ssh to ssh connection, ssh key, remote ssh

3. Duplicate Triggers Across Skills (30+ conflicts)

Most problematic duplicates:

Trigger Used By Issue
git bitbucket, github, gitlab, vulnerability-remediation/github 4-way conflict + false positives
/codereview skills/code-review, plugins/pr-review/skills/code-review Ambiguous which activates
/release-notes 3 different locations Triple duplication
/sdk skills/openhands-sdk, plugins/openhands/skills/openhands-sdk Duplication
All openhands triggers skills/ AND plugins/openhands/skills/ Intentional? Causes confusion

Recommendation:

  • Remove git from all skills OR make each one specific
  • Deduplicate skills that exist in both skills/ and plugins/*/skills/

4. Missing name Field (1 skill)
  • plugins/magic-test/skills/magic-word - missing required name field

🟠 MODERATE ISSUES

5. Skills with ONLY Slash Triggers (no keyword triggers)

These 13 skills cannot be discovered through natural language - they require users to know the exact slash command:

  • skills/agent-creator - only /agent-creator
  • skills/agent-memory - only /remember
  • skills/agent-sdk-builder - only /agent-builder
  • skills/code-review - only /codereview, /codereview-roasted
  • skills/code-simplifier - only /simplify
  • skills/github-pr-review - only /github-pr-review
  • skills/iterate - only /iterate, /verify, /babysit
  • skills/release-notes - only /release-notes
  • plugins/openhands - only /openhands-cloud
  • plugins/pr-review/skills/code-review - only slash commands
  • plugins/pr-review/skills/github-pr-review - only /github-pr-review
  • plugins/release-notes - only /release-notes
  • plugins/release-notes/skills/release-notes - only /release-notes

Recommendation: Add keyword triggers alongside slash commands:

  • code-review: add review code, code quality, pr feedback
  • iterate: add fix ci, address review, merge ready
  • release-notes: add changelog, release notes, what changed

6. Inconsistent Naming Convention
  • skills/azure-devops uses azure_devops (underscore) while all other triggers use hyphens

Recommendation: Change to azure-devops for consistency.


🟡 MINOR ISSUES

7. Very Short Triggers (High False Positive Risk)

Due to substring matching, these are problematic:

Trigger Length Risk
uv 2 chars HIGH - matches many common words
npm 3 chars Medium - less common substring
ssh 3 chars Medium - could match edge cases
k8s 3 chars Low - uncommon substring
8. File Extensions as Triggers

These may be intentional for project-type detection:

  • deno.json, deno.jsonc, deno.lock
  • uv.lock

Note: These are actually safer than short words because they include the dot.

9. Descriptions Missing "Use When" Guidance (14 skills)

These descriptions do not clearly explain activation context:

  • plugins/cobol-modernization/skills/build-setup
  • plugins/cobol-modernization/skills/cobol-modernization-overview
  • plugins/migration-scoring/skills/* (most)
  • skills/github-pr-review
  • skills/iterate
  • skills/spark-version-upgrade
  • skills/theme-factory

Recommended Trigger Patterns

Based on how the SDK's substring matching works:

Instead of Use
git github repository, github repo, git clone
uv uv python, uv package manager, uv.lock
azure azure devops, azure repos
ssh ssh connection, ssh key, remote ssh
ticket linear ticket, linear issue
Single words Multi-word phrases (more specific)

This issue was created by an AI agent (OpenHands) on behalf of the user.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with openhands-sdk/openhands/sdk/skills/skill.py to confirm the substring-matching behavior, then inspect the SKILL.md frontmatter under skills/ and plugins/. Address the listed missing, generic, duplicate, and malformed triggers or names, and verify that natural-language activation is specific without breaking the documented slash commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.