E3SM-Project / E3SM-Project/simboard
[DevOps]: Optimize DB-backed ingestion state queries for large performance archives
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 22m
- Merged PRs (30d)
- 12
Description
## Summary
Coordinate this work with #261.
PRs #260 and #272 reduced repeated archive processing, but the ingestor still requests the full history for a machine. As that history grows, the request may become slower and larger than needed.
## Goal
Use the results of the incremental scan from #261 to request only the ingestion state needed for new or changed archive content.
## Coordination with #261
- #261 should be implemented first and identify new or changed archive paths.
- This issue should use those paths to limit the state requested from SimBoard.
- Keep the implementation work separate so scanning and state-query changes can be tested and reviewed independently.
- Coordinate the shared information passed between the two steps.
## Acceptance criteria
- Normal incremental runs request only relevant ingestion state.
- Full rescans and checkpoint resets remain correct.
- Existing ingestors remain supported during the transition.
- Measure request size and runtime before and after the change.
- Document how this issue works with #261.
## Scope
This issue covers database query and response efficiency. Filesystem traversal and checkpoints are covered by #261.
Contributor guide
Assessment
This issue has not been assessed yet.