stacklok / stacklok/toolhive

Workload API edit endpoint fails to spin up workload if secrets are included

Open
#1,616 1 comment 0 reactions 1 assignee View on GitHub

@lujunsan is already working on this.

Since Sep 1, 2025.

authentication bug
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Bug description

Calling POST /api/v1beta/workloads/{name}/edit returns 200 OK but the workload is never brought back up. After the edit call:

  • The workload either disappears from the list or remains stuck in starting state.
  • A subsequent edit attempt for the same workload returns Workload not found.

Steps to reproduce

  1. Create a workload (example uses the GitHub MCP server) with a secret mapping from a stored secret to the expected env name:

    POST /api/v1beta/workloads
    Content-Type: application/json
    
    {
      "name": "github",
      "image": "ghcr.io/github/github-mcp-server:v0.12.1",
      "transport": "stdio",
      "env_vars": {},
      "secrets": [
        { "name": "GITHUB_PERSONAL_TOKEN", "target": "GITHUB_PERSONAL_ACCESS_TOKEN" }
      ],
      "cmd_arguments": [],
      "network_isolation": false,
      "volumes": []
    }
    
    • Ensure the secret GITHUB_PERSONAL_TOKEN exists in the secret store.
  2. Verify the workload exists:

    GET /api/v1beta/workloads/github
    

    Example response (abridged):

    {
      "image": "ghcr.io/github/github-mcp-server:v0.12.1",
      "host": "127.0.0.1",
      "env_vars": { "MCP_TRANSPORT": "stdio" },
      "secrets": [
        { "name": "GITHUB_PERSONAL_TOKEN", "target": "GITHUB_PERSONAL_ACCESS_TOKEN" }
      ],
      "transport": "stdio",
      "permission_profile": { "name": "network", "network": { "outbound": { "insecure_allow_all": true } } },
      "proxy_mode": "sse",
      "network_isolation": false,
      "name": "github"
    }
    
  3. Call edit without changing anything meaningful (same image, same secrets, etc.):

    POST /api/v1beta/workloads/github/edit
    Content-Type: application/json
    
    {
      "image": "ghcr.io/github/github-mcp-server:v0.12.1",
      "transport": "stdio",
      "target_port": 0,
      "cmd_arguments": [],
      "env_vars": { "MCP_TRANSPORT": "stdio" },
      "secrets": [
        { "name": "GITHUB_PERSONAL_TOKEN", "target": "GITHUB_PERSONAL_ACCESS_TOKEN" }
      ],
      "network_isolation": false,
      "volumes": []
    }
    
  4. Observe the response and state:

    • The API returns 200 OK.
    • The workload does not come back up; it disappears from the UI/list or stays indefinitely in starting.
    • Calling POST /api/v1beta/workloads/github/edit again now yields Workload not found.

Expected behavior

  • POST /workloads/{name}/edit should apply changes and restart/bring back up the workload (or at least keep it present and transition to a healthy state).
  • The endpoint should not return 200 OK if the workload is torn down and not successfully recreated.

Actual behavior

  • API returns 200 OK, but the workload is not restarted and disappears or remains stuck in starting.
  • Subsequent edits return Workload not found.

Environment (if relevant)

  • OS/version: (local dev environment; reporter can fill—e.g., macOS/Linux)
  • ToolHive version: v0.2.13
  • Workload image: ghcr.io/github/github-mcp-server:v0.12.1

Additional context

  • Attempting the same edit flow via UI vs direct server (thv serve) shows the same behavior; the workload vanishes after a successful edit response.
  • Changing the workload name via edit is intentionally blocked (by design), so not part of this issue.
  • Repro was confirmed by two users; both observed the 200 followed by the workload not returning.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.