Startup appears to hang indefinitely while synchronously migrating thousands of legacy session pickles

Startup appears to hang indefinitely while synchronously migrating thousands of legacy session pickles

Description

After updating Code Puppy to 0.0.728, the application no longer reached the interactive prompt when using an existing, large ~/.code_puppy home.

Startup consistently stopped after:

You're on the latest version (0.0.728)
Current version: 0.0.728
Core plugins version: 0.0.8

There was no error message, progress indicator, or explanation of ongoing work.

The application started normally with a fresh Code Puppy home.

Environment

• Code Puppy: 0.0.728
• Core plugins: 0.0.8
• OS: Windows
• Installation: uv tool
• Existing Code Puppy home containing a large number of legacy session files

Root cause

Immediately after printing the core plugins version, cli_runner.py synchronously runs:

sweep_contexts_to_autosaves()
sweep_legacy_pickle_sessions()

Only after both functions return does startup continue to plugin callbacks and the interactive prompt.

Relevant startup path:

# cli_runner.py
emit_system_message(core_plugins_message)

sweep_contexts_to_autosaves()
sweep_legacy_pickle_sessions()

await callbacks.on_startup()

The legacy format migration scans these directories:

autosaves/
autosaves/acp/
contexts/
subagent_sessions/

For every legacy .pkl without a JSON twin, it synchronously:

1. reads the complete pickle using Path.read_bytes(),
2. deserializes it,
3. normalizes the history,
4. validates the messages,
5. writes a JSON envelope,
6. moves the original into pre_v2_backup/.

No progress is emitted until the entire sweep finishes.

Scale observed

The existing home contained approximately:

autosaves/
  3,632 files
  3.72 GB total

subagent_sessions/
  23,266 files
  5.46 GB total

A previous interrupted migration had already archived approximately:

2,730 subagent pickle files

However, the migration marker had not yet been written, and the next startup still had:

5,540 pending subagent pickle files
754 MB of pending pickle data

There were also:

909 failed autosave pickles
1,332 failed subagent pickles
2,241 failed pickles total
~947 MB total

Additional problem: failed migrations are retried on every startup

_retry_quarantined() runs even when the main migration marker exists:

for directory in directories:
    dir_rescued, _stuck = _retry_quarantined(directory)

This means every startup can repeatedly read, deserialize, normalize, and validate all files under:

pre_v2_backup/failed/

In this case, that meant retrying more than 2,000 known-failing pickle files on every launch.

The comment currently describes this retry as “cheap”, but it is not cheap for large session stores:

Runs even when the sweep marker exists (cheap: only when failed/ is non-empty)

A non-empty failed/ directory can contain gigabytes of data and thousands of files.

Why this looks like a startup failure

The migration:

• blocks the main startup path,
• runs before the interactive prompt,
• does not show a pre-scan summary,
• does not show per-file or aggregate progress,
• does not provide an ETA,
• does not expose a cancel/defer option,
• may take a very long time,
• retries known failures on every subsequent launch.

As a result, Code Puppy appears frozen immediately after the version output.

No useful entry was written to errors.log, because this is not necessarily an exception or crash—the process is still busy performing synchronous migration work.

Expected behavior

Code Puppy should reach the interactive prompt within a bounded amount of time, even when the home contains a large legacy session archive.

A large migration should be one or more of:

• explicitly announced,
• progress-reporting,
• cancellable,
• resumable,
• time-budgeted,
• deferred until after startup,
• performed lazily when a session is accessed,
• exposed as an explicit maintenance command.

Known-failing files should not be retried without a bound on every startup.

Actual behavior

Code Puppy appears to hang after displaying the core plugins version and may spend an unbounded amount of time migrating or retrying thousands of pickle files before showing the prompt.

Interrupting the process leaves the global migration marker absent, causing the migration path to run again on the next startup.

Suggested fixes

1. Do not perform an unbounded migration synchronously during startup

Prefer one of:

code-puppy migrate-sessions

or:

• lazy migration when a session is loaded,
• background migration after the prompt is available,
• a bounded startup batch with continuation on later runs.

For example, migrate at most:

• N files,
• N megabytes,
• or N seconds

per startup.

2. Consider excluding legacy subagent sessions from mandatory startup migration

subagent_sessions/ can be much larger than user-facing autosaves and often contains disposable or historical internal agent runs.

Possible alternatives:

• migrate them lazily,
• archive them without conversion,
• provide a separate opt-in migration command,
• prune them according to an explicit retention policy.

User-facing named sessions and internal subagent artifacts should not necessarily have identical startup migration requirements.

3. Persist granular migration progress

The current global marker is only written after every directory completes:

for directory in directories:
    _migrate_directory(directory)

marker.touch()

Use a migration journal or per-directory checkpoints instead, for example:

{
  "version": 2,
  "directories": {
    "autosaves": "complete",
    "autosaves/acp": "complete",
    "contexts": "complete",
    "subagent_sessions": {
      "state": "in_progress",
      "last_processed": "..."
    }
  }
}

The existing per-file JSON check provides some idempotency, but it does not prevent a large directory from being rescanned and does not explain progress to the user.

4. Do not retry all quarantined failures on every startup

Retries should happen only when one of these conditions applies:

• the session migration/unpickler implementation version changed,
• the user explicitly runs a retry command,
• a bounded retry budget is available,
• a per-file retry count or backoff permits it.

For example:

code-puppy migrate-sessions --retry-failed

Store metadata such as:

{
  "attempts": 2,
  "last_attempt_version": "0.0.728",
  "last_error": "...",
  "last_attempt_at": "..."
}

A file that already failed under the current migration implementation should not be retried on every launch.

5. Show progress before processing files

At minimum, emit something like:

Legacy session migration required.
Found 5,540 sessions (754 MB).
Progress: 312 / 5,540
Press Ctrl+C to defer migration and continue startup.

The initial scan should happen before expensive deserialization so the user can understand why startup is delayed.

6. Add resource and disk-space safeguards

Before migration:

• count files,
• estimate total input size,
• estimate required output space,
• check available disk space,
• avoid loading unnecessarily large files without limits,
• ensure interruption leaves each individual migration atomic.

7. Add scale and interruption tests

Suggested tests:

• 10,000 legacy session files,
• multi-gigabyte aggregate input,
• thousands of quarantined failures,
• interruption halfway through a directory,
• restart after interruption,
• insufficient disk space,
• migration of user sessions while deferring subagent sessions,
• bounded startup duration.

Workaround

A non-destructive workaround was to move legacy data out of the exact directories scanned at startup:

subagent_sessions/
    -> _subagent_sessions_legacy_backup/

autosaves/pre_v2_backup/failed/
    -> autosaves/pre_v2_backup/failed_legacy_backup/

No files were deleted.

On the next startup, the migration found no pending files in the scanned locations, wrote the migration marker, and Code Puppy reached the interactive prompt normally.

Relevant files

code_puppy/cli_runner.py
code_puppy/session_format_migration.py
code_puppy/session_migration.py

The most relevant functions are:

sweep_legacy_pickle_sessions()
_migrate_directory()
migrate_pickle_file()
_retry_quarantined()

Summary

This is not primarily a migration correctness problem. It is a startup architecture and UX problem:

An unbounded, potentially multi-gigabyte migration must not silently block the interactive application startup.

The retry behavior for quarantined files makes the issue persistent even after the main migration has otherwise completed.

Start by reading code_puppy/cli_runner.py and code_puppy/session_format_migration.py, then trace sweep_legacy_pickle_sessions(), _migrate_directory(), migrate_pickle_file(), and _retry_quarantined(). Review the existing migration tests, if present, and add coverage for large stores, interruption, restart, and quarantined failures. Done should mean startup is bounded and explains or defers migration without retrying known failures indefinitely.