stacklok / stacklok/toolhive

Switch status writes from Update to Patch across all controllers

Open
#4,633 3 comments 0 reactions 1 assignee View on GitHub

@jhrozek is already working on this.

Since Apr 20, 2026.

bug go kubernetes operator
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Summary

Every controller uses r.Status().Update(ctx, obj) which replaces the entire status subresource. Concurrent writers overwrite each other's changes. This is especially dangerous for VirtualMCPServer where the runtime process and controller write to different status fields simultaneously.

Severity: MUST FIX
Area: Controller Logic
Breaking: No

Location

  • Every controller file under cmd/thv-operator/controllers/
  • 35 Status().Update call sites in mcpserver_controller.go alone
  • Zero Status().Patch calls found anywhere in the codebase

Problem

r.Status().Update(ctx, obj) replaces the entire status subresource, meaning concurrent writers overwrite each other's changes. This is especially dangerous for VirtualMCPServer where the runtime process writes status.discoveredBackends, status.backendCount, and BackendsDiscovered condition while the controller writes infrastructure conditions.

Impact

  • Concurrent status updates cause data loss (last writer wins)
  • VirtualMCPServer runtime and controller can overwrite each other
  • Conflict errors under load leading to unnecessary requeues

Recommended Fix

  1. Switch all status writes to use client.MergeFrom patch:
    patch := client.MergeFrom(obj.DeepCopy())
    // ... mutate status ...
    r.Status().Patch(ctx, obj, patch)
    
  2. Start with VirtualMCPServer (highest risk due to concurrent runtime writes)
  3. Then MCPServer (highest volume — 35 call sites)
  4. Then all other controllers

Related Issues

  • Batch MCPServer status updates to reduce API calls per reconciliation (#4630)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.