E3SM-Project / E3SM-Project/simboard
[DevOps]: v3 data diagnostics backfill
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 25m
- Merged PRs (30d)
- 14
Description
## Task Details
After #281 is complete and the E3SM v3 data has been ingested, backfill the corresponding diagnostic links.
Create a one-time diagnostic-backfill script using the current v3 ingestion script as its structural basis for the static target manifest, reconciliation, and reporting. Adapt it to scan diagnostic output, construct scanner-compatible provenance metadata, archive diagnostics, and link them to the already ingested SimBoard cases.
Use the diagnostic-source mapping in the issue comment.
## Machine-scoped execution
The backfill must accept a required machine-selection flag and process only targets for that machine:
- **Chrysalis**: all mapped v3 targets other than `v3.LR.amip_bonus_0101`.
- **Perlmutter**: `v3.LR.amip_bonus_0101`.
The script must reject an unsupported machine, and its reconciliation output must identify the selected machine and targets skipped because they belong to another machine. Although `v3.LR.amip_bonus_0101` ran on Perlmutter, use the diagnostic output for `v3.LR.amip_0101` under ac.wlin: the runs are bit-for-bit identical and the bonus run only adds output.
## Case identity, archive layout, and generated provenance settings
The original diagnostic directories do not contain enough information to recover their original provenance settings. Recover the scanner-required values from the authoritative, already ingested SimBoard case record instead.
For every static target:
1. Use the selected machine to resolve the expected `machine_id`.
2. Query SimBoard by exact `case_name` and `machine_id`.
3. Require exactly one matching case; fail and report zero or multiple matches.
4. Derive `hpc_username` and `case_group` from that case record. `case_group` is nullable: if it is absent, use an ungrouped archive layout and omit `case_group` from generated settings.
5. Stage diagnostics at `/diagnostics_archive/production///`.
6. Find the latest existing `provenance..cfg` in the diagnostic output directory. Do **not** create or replace this file.
7. Create the matching `provenance..settings` file using the timestamp from that latest `.cfg` file.
8. Write these settings fields:
- `case_name`
- `machine`
- `hpc_username`
- `case_group` when non-null
- `diagnostics_url`
For example, a Chrysalis case with `case_group=v3.LR` is staged at:
```
/lcrc/group/e3sm/public_html/diagnostic_output/diagnostics_archive/production/v3.LR/v3.LR.historical_0051/provenance.20260805_234735_275469.settings
```
Its generated settings contain:
```
case_name = v3.LR.historical_0051
machine = chrysalis
hpc_username = ac.wlin
case_group = v3.LR
diagnostics_url = https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/diagnostics_archive/production/v3.LR/v3.LR.historical_0051
```
Construct `diagnostics_url` from the staged archive path:
- Chrysalis LCRC: `https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/diagnostics_archive/production//`
- Perlmutter NERSC: `http://portal.nersc.gov/cfs/e3sm/diagnostics_archive/production//`
The final scanner/API link must resolve the exact tuple: `case_name`, `machine_id`, and `hpc_username`. A name-only lookup is diagnostic-only and must never select a case automatically.
## Steps to Complete
1. Select static targets for the requested machine and their mapped diagnostic sources.
2. Locate and copy each matching diagnostic directory into the database-derived archive layout.
3. Resolve each target to exactly one existing SimBoard case, find its latest provenance `.cfg`, and generate the matching scanner-compatible `.settings` file.
4. Run the SimBoard diagnostic scanner.
5. Verify the scanner links each diagnostic directory to the correct case using `case_name`, `hpc_username`, and `machine_id`.
6. Emit a reconciliation report for copied, linked, missing, unmapped, ambiguous, and machine-skipped targets.
## Acceptance Criteria
- [ ] The script requires a supported machine-selection flag and processes only that machine’s targets.
- [ ] Chrysalis runs process all mapped Chrysalis targets; Perlmutter runs process `v3.LR.amip_bonus_0101` using the mapped, bit-for-bit-identical `v3.LR.amip_0101` diagnostics.
- [ ] Each target resolves to exactly one SimBoard case by `case_name` and `machine_id`, and derives `hpc_username` and `case_group` from that record.
- [ ] A null `case_group` produces an ungrouped archive layout; a non-null value produces the matching grouped layout.
- [ ] The script preserves the latest existing provenance `.cfg` and generates only the same-timestamp `.settings` file with the scanner-required fields.
- [ ] Generated `diagnostics_url` values use the correct public archive base for Chrysalis or Perlmutter.
- [ ] The scanner links each copied diagnostic directory to the correct ingested case.
- [ ] The reconciliation report documents copied, linked, missing, unmapped, ambiguous, and machine-skipped targets.
Contributor guide
Research direction
Start with the current v3 ingestion script to understand its static manifest, reconciliation, and reporting structure, then locate the SimBoard diagnostic scanner entry point. Use the diagnostic-source mapping from the issue comment and verify case resolution, archive staging, generated settings, scanner links, and machine-specific reconciliation output. Done means all acceptance criteria pass for both machine selections without replacing existing provenance .cfg files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100