lablup / lablup/backend.ai

Full pickle removal + list_instances()-based restart recovery; report dangling as events

Open
#12,619 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Pickle is NOT used at all. Delete kernel-registry pickle serialization (kernel_registry/writer/pickle.py, loader/pickle.py, pickle/creator.py) and the pickle_writer/loader wiring in docker/agent.py; convert kubernetes_recovery.py (still pickle-based) to container/scratch-JSON. Recovery uses only the existing container-based path (writer/container.py stores recovery JSON in scratch; loader/container.py rebuilds by enumerating live containers). Each manager correlates list_instances() on kernel id - alloc from device bindings (reconstruct_resource_usage), ports from port bindings (_restore_ports). LifecycleService reports orphan/dangling to the Manager as events and OrphanKernelCleanupObserver is deleted. release is idempotent so recovery re-invocation is safe. Recovering directly from live instances also removes the need for a pickle-reading migration shim.

Separate pickle path: agent.py kconfig.dat (per-kernel restart config, pickle.dumps/loads) is also pickle and a replacement target, but restart_kernel is not Sokovan-driven and is kernel-internal, so confirm the BA-6074/BEP-1002 boundary first (flag here; split into its own story if needed).

Success Criteria:
- [ ] No 'import pickle' remains in agent/kernel_registry/ (writer/loader/creator deleted).
- [ ] After agent restart, alloc/port state is recovered from live instances (no pickle; container/scratch-JSON).
- [ ] kubernetes_recovery no longer uses pickle (container/scratch based).
- [ ] Dangling (registered-alive) reported as Manager events, not silent cleanup.
- [ ] Recovery is idempotent under repeated restart.
- [ ] pants test passes for affected packages

JIRA Issue: BA-6765

Contributor guide

Open the contributing guide

Research direction

Start by reading kernel_registry/writer/pickle.py, loader/pickle.py, pickle/creator.py, writer/container.py, loader/container.py, docker/agent.py, and kubernetes_recovery.py. Confirm the BA-6074/BA-1002 boundary for kconfig.dat before splitting that work. Done means recovery uses container scratch-JSON and live instances, reports dangling kernels as Manager events, remains idempotent, and affected pants tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, python
Domain
backend, distributed-systems, infrastructure
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.