CopilotKit / CopilotKit/outpost
Stop the cutover and rollback scripts reporting success while shadow mode is unresolved
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
scripts/cutover/execute-cutover.ts and scripts/cutover/rollback.ts are the two scripts that flip shadow mode around a cutover. Both report the step succeeded without confirming the flag actually moved, and in opposite directions from each other.
The cutover path passes while shadow mode is still on
disableShadowMode has two exits and both are PASS. The second one is reached precisely when shadow mode is still engaged:
console.log(' ACTION REQUIRED: Set SHADOW_MODE=false in Railway environment variables');
console.log(' Then redeploy the discord-bot service.');
return { step, status: 'PASS', message: 'Shadow mode disable instruction issued', timestamp: new Date() };
Steps 1-3 in executeCutover each gate on status === 'FAIL' and return early. Step 4 pushes its result and falls straight through to the announcement step and log.outcome = 'SUCCESS'.
So with the flag still on, --confirm prints:
[OK] Disable shadow mode: Shadow mode disable instruction issued
[OK] Post announcement: Announcement: "..."
Cutover COMPLETED
and exits 0. The announcement goes to the community, the incumbent gets turned off, and Outpost answers nobody — because shadow mode never came off.
Also, currentValue is interpolated into the already-disabled message but dropped from the still-on one. The branch where the raw value is the whole diagnosis is the branch that omits it.
The rollback path never reads the flag at all
enableShadowMode prints three ACTION REQUIRED lines and returns status: 'DONE'. It does not call isShadowMode(), does not read process.env.SHADOW_MODE, and verifies nothing. executeRollback gates only step 1, so the outcome is SUCCESS regardless.
This is the worse half, because it is the incident path. Production is double-posting, the operator runs rollback --confirm, reads Rollback COMPLETED, and moves on to the 30-minute monitoring step while Outpost keeps posting at real reporters the whole time.
Suggested shape
Make the unresolved branch not a pass, carry the raw value, and gate the step the way the three above it are gated:
return {
step,
status: 'FAIL',
message:
`Shadow mode is still ON (SHADOW_MODE=${currentValue ?? 'unset'}). ` +
'Set it to false in Railway, redeploy, and re-run — cutover is not complete while the flag is engaged.',
timestamp: new Date(),
};
If a manual step reading as a hard failure is wrong, a third status (ACTION_REQUIRED) treated as non-SUCCESS in the aggregation works too. "PASS / COMPLETED" for a step that just verified the flag is still on is the part that needs to go.
The rollback fix mirrors it, and isShadowMode() helps there: an unrecognized value reads as ON, which is the right answer for a rollback.
Test gap that goes with it
scripts/__tests__/cutover.test.ts covers runHealthChecks, verifyTicketData, executeCutover and executeRollback, but there is no test for disableShadowMode and nothing in the file sets SHADOW_MODE. Worth adding the same fence the queue suites got in #233 — it.each(['TRUE', '1', 'yes', 'on', ' true ', '', 'flase']) asserting the step does not report the cutover done.
Use the delete-when-absent restore from packages/outpost/queue/src/__tests__/ai-response.test.ts: assigning undefined stores the string "undefined", which is an unrecognized value under the current rules and would leak shadow-mode-ON into every later test in the file.
Both behaviours predate #233 — that PR routed the comparison through isShadowMode(), which fixed which direction the message is wrong in and is what made this visible. Filed separately so that diff stays about the predicate.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/cutover/execute-cutover.ts and scripts/cutover/rollback.ts, tracing disableShadowMode, enableShadowMode, and their outcome gates. Then read scripts/tests/cutover.test.ts and the delete-when-absent restore pattern in packages/outpost/queue/src/tests/ai-response.test.ts. Done means unresolved or unrecognized SHADOW_MODE values cannot report cutover or rollback success, with coverage for the listed values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- devops, release
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100