E3SM-Project / E3SM-Project/simboard

[DevOps]: Optimize DB-backed ingestion state queries for large performance archives

Open
#205 3 comments 0 reactions 0 assignees View on GitHub
type: devops
Dominant language
Python
Stars
1
Forks
2
Avg merge
22m
Merged PRs (30d)
12

Description

## Summary

Coordinate this work with #261.

PRs #260 and #272 reduced repeated archive processing, but the ingestor still requests the full history for a machine. As that history grows, the request may become slower and larger than needed.

## Goal

Use the results of the incremental scan from #261 to request only the ingestion state needed for new or changed archive content.

## Coordination with #261

- #261 should be implemented first and identify new or changed archive paths.
- This issue should use those paths to limit the state requested from SimBoard.
- Keep the implementation work separate so scanning and state-query changes can be tested and reviewed independently.
- Coordinate the shared information passed between the two steps.

## Acceptance criteria

- Normal incremental runs request only relevant ingestion state.
- Full rescans and checkpoint resets remain correct.
- Existing ingestors remain supported during the transition.
- Measure request size and runtime before and after the change.
- Document how this issue works with #261.

## Scope

This issue covers database query and response efficiency. Filesystem traversal and checkpoints are covered by #261.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.