spacedriveapp / spacedriveapp/spacebot

[Awareness] Idle Spacebot Consumed Entire Monthly Token Plan in ~2 Days — Cortex Timer-Driven Synthesis

Open
#593 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
2.4k
Forks
367
PR merge metrics
No merged PRs in 30d

Description

TL;DR

Spacebot was installed for practice/testing. With zero active usage (only 2 chat inputs over 11 days), the default cortex tick interval of 60s burned through a $100–$200 monthly token subscription in approximately a few days. The token waste came from timer-driven synthesis running on unchanged memories/conversations.


Environment

  • Spacebot version during incident: v0.4.x (previous version)
  • Spacebot version now: v0.5.0
  • Config: Mostly defaults, cortex tick_interval_secs = 60
  • Usage pattern: Installed for practice, not actively used

What Happened — Timeline

Date Event
Apr 27 Spacebot deployed
Apr 29 Brief usage (2 chat messages + 1 worker run)
Apr 30 – May 6 Zero user activity
May 7 API key returns 401 Unauthorized — limit exhausted
May 8 Investigation reveals ~2B tokens consumed while idle

Root Cause: Timer-Driven Synthesis

The default [defaults.cortex] configuration:

[defaults.cortex]
tick_interval_secs        = 60   # Every 60 seconds
worker_timeout_secs       = 1800
branch_timeout_secs       = 120
circuit_breaker_threshold = 3

On each tick, cortex attempts:

  1. generate_knowledge_synthesis
  2. generate_bulletin
  3. generate_profile
  4. Association backfill

Even with zero new activity, each tick re-processes the full agent context.

Result:

  • ~2B tokens consumed over ~11 days of idle operation
  • Provider's monthly subscription usage limit fully exhausted
  • Quick, simple setup — no active usage, no new memories or conversations

Impact

  • Financial: Consumed entire monthly token plan ($100–$200 estimated)
  • Operational: API key invalidated, Spacebot degraded to unusable state
  • Trust: User almost uninstalled after thinking it was a bug/misconfiguration

Suggestions (Non-Blaming)

1. Warning/Documentation

A prominent warning in docs would help:

⚠️ Default cortex interval is 60s. For metered/token-limited plans, increase to 86400 (daily) or disable until actively using Spacebot.

2. Configuration Guide

A dedicated doc for cortex configuration:

# Conservative (metered plans)
[defaults.cortex]
tick_interval_secs = 86400  # Daily

# Moderate (moderate usage)
[defaults.cortex]
tick_interval_secs = 3600   # Hourly

# Aggressive (unlimited plans)
[defaults.cortex]
tick_interval_secs = 300    # Every 5 min
3. Token Estimation

Could cortex estimate tokens per tick based on memory count and warn if it exceeds a threshold?

4. Idle Detection

Skip synthesis if no memories/conversations changed since last tick.


What's Already in v0.5.0 vs What's Planned

Note: The token consumption happened on v0.4.x (previous version). We updated to v0.5.0 after discovering the issue.

Shipped in v0.5.0 (great additions that helped us investigate):

  • token_usage table — tracks every API call with input/output tokens
  • cortex_events table — logs every synthesis attempt with errors
  • ✅ Config watcher — reloads config without restart
  • ✅ Working memory tables — foundation for the redesign

Still NOT in v0.5.0 (the root cause remains):

  • Change-driven synthesis — no version counter or skip logic found in DB schema or binary
  • ❌ Idle detection — cortex still runs on every tick regardless of changes

I see the working memory implementation plan (Phase 3) addresses this with change-driven synthesis:

let current_version = knowledge_synthesis_version.load();
let last_version = knowledge_synthesis_last_version.load();
if current_version == last_version {
    return;  // Skip - nothing changed!
}

Status check: The plan doc is ~3 weeks old. Is Phase 3 (change-driven synthesis) still in progress? Is there a timeline or branch we can follow?


Our Mitigation (Applied)

We changed the config to:

[defaults.cortex]
tick_interval_secs = 86400  # Once per day

This reduces consumption in my providers.

Also reconfigured routing to use a primary model for conversations and a cheaper model for background tasks.


Questions for the Team

  1. Is there an ETA for Phase 3 (change-driven synthesis)?
  2. Should the default tick_interval_secs be higher for new installations?
  3. Would a "first-run wizard" that asks about token plans be helpful?
  4. Can we help test or contribute to the working memory redesign?

Thank you for v0.5.0


Labels: awareness, documentation, cortex, token-usage, configuration

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/design-docs/working-memory-implementation-plan.md and the [defaults.cortex] configuration described in the report. Determine which warning, configuration guidance, or Phase 3 status update is intended, then verify the result against the documented v0.5.0 behavior. Done means the scope is reduced to a specific documentation change with clear guidance for metered plans.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.