lablup / lablup/backend.ai

Dual-write installed images to agent_images on heartbeat (bulk upsert + diff-delete + exit cleanup)

Open
#13,413 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Extend sync_installed_images (repositories/agent/repository.py) to write the agent_images DB table in addition to the existing Valkey set writes (Valkey writes kept for rollback safety). Reuses the ImageIdentifier list and the resolved images_data already produced in this method.

Scope:
- Bulk upsert into agent_images with conflict target on the (agent_id, image_id) unique constraint via execute_bulk_upserter index_elements; ON CONFLICT updates updated_at; the surrogate id is excluded from insert values and generated by the column default.
- Diff-delete: rows of the agent whose image_id is not in the reported list are deleted via a BatchPurgerSpec subquery (agent_id equals the reporting agent AND image_id NOT IN the reported set).
- Agent exit (mark_agent_exit): delete agent_images rows of that agent, alongside the existing Valkey cleanup.
- The AgentInstalledImagesRemoveEvent path needs no DB counterpart: diff-delete absorbs scan-detected removals.

Success Criteria
- [ ] heartbeat inserts new rows and refreshes updated_at for existing rows
- [ ] an image removed on the agent disappears from agent_images on the next heartbeat (diff-delete)
- [ ] agent exit removes all agent_images rows of that agent
- [ ] canonicals not registered in the images table are skipped silently (same as current behavior)
- [ ] Valkey writes still occur unchanged (rollback safety)
- [ ] pants test passes for affected packages

JIRA Issue: BA-7164

Contributor guide

Open the contributing guide

Research direction

Start in repositories/agent/repository.py, tracing sync_installed_images, its existing Valkey writes, the ImageIdentifier list, and resolved images_data; then inspect mark_agent_exit for the exit cleanup path. Implement and test the agent_images bulk upsert, diff-delete, and exit deletion while preserving Valkey behavior, then run pants test for the affected packages.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.