zai-org / zai-org/feedback

[Bug] AutoClaw v1.17.2: Hermes Evolution Board Card Decryption Failure & Evaluator Silent After Reset

Open
#298 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority: P2
Dominant language
No language data
Stars
22
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Bug Report: AutoClaw Hermes Evolution Board β€” Dual Failure Mode

πŸ“‹ Summary

Two critical bugs in AutoClaw v1.17.2 Hermes Evolution Board card flow:

  1. Card Decryption Dead-End β€” Evolution cards become permanently undecryptable after app-data reset/Keychain rotation, causing both Confirm and Reject buttons to fail
  2. Evaluator Silent After Reset β€” Post-turn evolution evaluator stops generating any cards entirely after manual app-data reset

Severity: HIGH β€” Core learning/memory feature completely broken


πŸ–₯️ Environment

Field Value
App AutoClaw v1.17.2 (bundle com.zhipuai.autoclaw)
OS macOS (Darwin 24.6.0, arm64, MacBook Pro)
Node.js v22.22.3
Timezone Europe/Paris
Date of Incident 2026-08-16
Reporter Baku159 (Discord)
Discord Thread https://discord.com/channels/1346756824233148527/1538882180745265213

πŸ› Issue #1: Card Cannot Be Decrypted (Confirm/Reject Both Fail)

What Happened

An Evolution Board card was generated proposing a MEMORY.md entry ("save the welder project memory"). Clicking Confirm produced a decryption error. Clicking Reject also produced a decryption error. The card was effectively stuck β€” neither action could complete.

Root Cause (Diagnosed)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    KEY MISMATCH SCENARIO                     β”‚β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                             β”‚β”‚  Timeline:                                                   β”‚β”‚  ─────────                                                    β”‚β”‚  T1: Card generated + encrypted with Key-A                   β”‚β”‚      β†’ Card payload sealed with safeStorage encryption       β”‚β”‚      β†’ Key stored in macOS Keychain:                         β”‚β”‚        Service: "autoclaw Safe Storage"                     β”‚β”‚        Account: "autoclaw"                                  β”‚β”‚                                                             β”‚β”‚  T2 (~12:42): App-data profile recreated/restored            β”‚β”‚      β†’ Old profile (with encrypted card) restored            β”‚β”‚                                                             β”‚β”‚  T3 (~12:44): Keychain entry ROTATED                        β”‚β”‚      β†’ New Key-B created in Keychain                        β”‚β”‚      β†’ Key-A (original) destroyed/overwritten               β”‚β”‚                                                             β”‚β”‚  T4 (User tries Confirm/Reject):                             β”‚β”‚      β†’ App attempts decrypt with Key-B                       β”‚β”‚      β†’ Card was encrypted with Key-A ❌                      β”‚β”‚      β†’ CIPHERTEXT PERMANENTLY UNDECRYPTABLE                  β”‚β”‚      β†’ Both buttons throw decryption error                   β”‚β”‚                                                             β”‚β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜```

### Evidence Collected

| Timestamp (Europe/Paris) | Action | Result |
|--------------------------|--------|--------|
| ~18:09 | Confirm attempt | ❌ Decryption error |
| ~19:05 | Confirm attempt | ❌ Decryption error |
| ~19:08 | Reject attempt | ❌ Decryption error |

**Backup preserved for inspection:**
`~/Library/Application Support/autoclaw.backup-20260816`
(Contains the undecryptable card state)

**Keychain state:**
- Entry `autoclaw Safe Storage` (account: `autoclaw`) exists
- Key creation time coincides with profile restore window (~12:44)
- Original encryption key (pre-T3) no longer available

### Impact

- User cannot confirm or reject evolution proposals
- Card stuck in UI indefinitely
- No discard/recovery path available
- Memory/learning feature blocked

---

## πŸ› Issue #2: Evaluator Produces No Cards After Manual Reset

### What Happened

Because neither Confirm nor Reject worked on the stuck card, user performed manual reset:

1. Renamed app data β†’ `~/Library/Application Support/autoclaw.backup-20260816`
2. Fresh profile created (~19:28)
3. Fresh Keychain key generated (`autoclaw Safe Storage`, created 19:28)

**Result:** Decryption bug eliminated (old card gone), BUT:
- Post-turn evolution evaluator **stopped producing cards entirely**
- Repeating qualifying request ("save the welder project memory") 3Γ— produced **zero cards**

### Evidence Collected

| Timestamp (Europe/Paris) | Action | Result |
|--------------------------|--------|--------|
| ~19:32 | Qualifying request #1 | ❌ No card generated |
| ~19:38 | Qualifying request #2 | ❌ No card generated |
| ~19:40 | Qualifying request #3 | ❌ No card generated |

**Technical Inspection:**

βœ… Checked fresh profile's IndexedDB and Local Storage:
   - No `proposal`, `evolution`, `hermes`, `welder`, or `memory` content present

βœ… IndexedDB write analysis:
   - Single generic write at 19:38 (`feilin_indexeddb_...`)
   - This was a routine record, NOT a proposal record
   - Confirms no card was persisted

❌ Missing system injection:
   - Session never received `[SYSTEM: Post-turn evolution check]`
   - This message normally authorizes evaluation
   - Its absence suggests evaluator was never triggered

### Suspected Cause

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ EVALUATOR GATE HYPOTHESIS β”‚β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β”‚β”‚ The post-turn evolution evaluator may be gated by: β”‚β”‚ β”‚β”‚ 1. Server-side flag (feature toggle) β”‚β”‚ β†’ May have been disabled/reset when app data wiped β”‚β”‚ β†’ No client-side control visible β”‚β”‚ β”‚β”‚ 2. App setting / preference β”‚β”‚ β†’ "Hermes / Evolution / learning" toggle β”‚β”‚ β†’ Reset to OFF during data wipe β”‚β”‚ β†’ NOT exposed in user-facing settings UI β”‚β”‚ β”‚β”‚ 3. Profile initialization state β”‚β”‚ β†’ Fresh profile missing required bootstrap data β”‚β”‚ β†’ Evaluator requires "priming" before first run β”‚β”‚ β”‚β”‚ 4. Session/context state β”‚β”‚ β†’ Evaluator tied to session lifecycle β”‚β”‚ β†’ Reset broke session-evaluator linkage β”‚β”‚ β”‚β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜```


❓ Questions for Development Team

Regarding Issue #1 (Decryption):
  1. Detection: Can the app detect key mismatch before attempting decryption?
  2. Recovery: Should there be a "Discard" path for undecryptable cards instead of dead-ending both actions?
  3. Migration: Is there a key versioning scheme that could support key rotation without data loss?
Regarding Issue #2 (Evaluator Silent):
  1. Flag Dependency: Does the post-turn evaluator depend on a server-side flag that may reset on data wipe?
  2. Toggle Existence: Is there a Hermes/Evolution/learning toggle (client or server) and how to re-enable it?
  3. Known Issue: Any known bug for Evolution Board cards not generating after app-data reset on v1.17.2?
  4. Bootstrap: Does the evaluator require initial setup/priming that doesn't happen automatically on fresh profiles?

πŸ”§ Suggested Fixes

For Issue #1 (Decryption Dead-End):
// PROPOSED: Graceful handling of undecryptable cards

async function handleCardAction(action: 'confirm' | 'reject', card: EncryptedCard) {
  try {
    const decrypted = await decrypt(card.payload);
    await processAction(action, decrypted);
  } catch (error) {
    if (error.code === 'DECRYPTION_FAILED') {
      // Offer recovery options instead of throwing
      showRecoveryDialog({
        title: 'Card cannot be decrypted',
        message: 'This card was encrypted with a previous key.',
        options: [
          { label: 'Discard card', action: () => discardCard(card.id) },
          { label: 'View raw (if possible)', action: () => inspectRawPayload(card) },
          { label: 'Report issue', action: () => reportBug(error, card) }
        ]
      });
    }
  }
}

Additional hardening:

  • Add key version identifier to encrypted payload
  • On app-data restore, detect key version mismatch upfront
  • Offer to discard all cards encrypted with unavailable keys
For Issue #2 (Evaluator Silent):
// PROPOSED: Evaluator health check & re-initialization

class EvolutionEvaluator {
  async initialize() {
    // Ensure required flags/settings exist
    await this.ensureServerFlags();
    await this.ensureLocalSettings();
    
    // Log initialization state for debugging
    this.logState({
      serverFlags: this.getServerFlags(),
      localSettings: this.getLocalSettings(),
      profileAge: this.getProfileAge()
    });
  }
  
  async checkShouldEvaluate(turnContext: TurnContext) {
    // Pre-flight checks
    if (!await this.isInitialized()) {
      console.warn('[Hermes] Evaluator not initialized, skipping');
      await this.reinitialize(); // Attempt auto-recovery
    }
    
    if (!this.isFeatureEnabled()) {
      console.warn('[Hermes] Feature disabled, skipping');
      return false;
    }
    
    return this.qualifiesForEvaluation(turnContext);
  }
}

Additional diagnostics:

  • Add visible indicator in Settings showing Evolution status (Active/Disabled/Error)
  • Log [SYSTEM: Post-turn evolution check] injection attempts (success/failure reason)
  • Emit diagnostic event when evaluator skips a qualifying turn

πŸ§ͺ Reproduction Steps (for QA)

Reproduce Issue #1 (Decryption Failure):
Prerequisites:
- AutoClaw v1.17.2 on macOS
- Working Evolution Board (can generate cards)

Steps:
1. Trigger a turn that produces an evolution proposal card
2. Note the current Keychain key (autoclaw Safe Storage)
3. Backup app-data, then simulate key rotation:
   - Delete/recreate Keychain entry (new key)
   - Or restore old app-data with new key in place
4. Attempt to Confirm or Reject the card

Expected (current buggy behavior):
- Both buttons throw decryption error
- Card stuck indefinitely

Desired (after fix):
- App detects key mismatch
- Offers discard/recovery option
- No unhandled exception
Reproduce Issue #2 (Evaluator Silent):
Prerequisites:
- AutoClaw v1.17.2 on macOS
- Known working Evolution Board (generates cards normally)

Steps:
1. Verify evaluator works: trigger qualifying turn β†’ card appears
2. Perform full app-data reset:
   - Quit AutoClaw completely
   - Rename/move: ~/Library/Application Support/autoclaw
   - Delete Keychain entry: autoclaw Safe Storage
3. Relaunch AutoClaw (creates fresh profile + key)
4. Trigger same qualifying turn (should produce card)
5. Repeat step 4 two more times

Expected (current buggy behavior):
- No [SYSTEM: Post-turn evolution check] injection
- Zero cards generated across all 3 attempts
- No error shown to user (silent failure)

Desired (after fix):
- Evaluator initializes properly on fresh profile
- Cards generate as expected
- If disabled, show clear indication in UI

πŸ“Š Impact Assessment

Metric Value
Severity HIGH β€” Core memory/learning feature broken
Affected Versions AutoClaw v1.17.2 (confirmed)
Affected Platform macOS (Keychain-dependent; other platforms may vary)
User Experience Data loss + feature silently breaks
Workaround Available ❌ None β€” full app reinstall doesn't guarantee fix
Regression Risk Medium β€” affects crypto + evaluator subsystems

πŸ™ Acknowledgments


Submitted by:
Regards,
Roman (Discord: bignavi_x)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up β€” it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file, test, or code entry point is identified; begin by reproducing the two failures using the supplied Keychain rotation and fresh-profile reset steps, then inspect the preserved backup and Keychain state. Done means undecryptable cards have a recovery path and a fresh profile reliably runs the post-turn evaluator or clearly reports why it is disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, typescript
Domain
desktop, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.