ISISComputingGroup / ISISComputingGroup/DataStreaming

`filewriter`: dynamic metadata

Offen
#88 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
filewriter
Vorherrschende Sprache
Keine Sprachdaten
Sterne
0
Forks
0
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

The filewriter must be able to write the 'dynamic' metadata for a run.

These are pieces of metadata which require knowledge from the `_events` (and/or other) streams to compute. They are 'dynamic' in the sense that they cannot be known at run start.

This includes:
- `raw_data_1/collection_time` (because `collection_time` excludes vetoed/paused time so needs per-frame veto flags)
- `raw_data_1/framelog/events_log`
- `raw_data_1/framelog/frame_log`
- `raw_data_1/framelog/good_frame_log`
- `raw_data_1/framelog/period_log`
- `raw_data_1/framelog/proton_charge`
- `raw_data_1/framelog/raw_events_log`
- `raw_data_1/good_frames`
- `raw_data_1/proton_charge`
- `raw_data_1/proton_charge_raw`
- `raw_data_1/raw_frames`
- `raw_data_1/runlog/count_rate`
- `raw_data_1/runlog/dae_beam_current`
- `raw_data_1/runlog/good_frames`
- `raw_data_1/runlog/good_uah_log`
- `raw_data_1/runlog/is_running`
- `raw_data_1/runlog/is_waiting`
- `raw_data_1/runlog/monitor_sum_1`
- `raw_data_1/runlog/np_ratio`
- `raw_data_1/runlog/period`
- `raw_data_1/runlog/period_change_log`
- `raw_data_1/runlog/raw_frames`
- `raw_data_1/runlog/raw_uah_log`
- `raw_data_1/runlog/run_status`
- `raw_data_1/runlog/total_counts`
- `raw_data_1/runlog/veto_log`
- `raw_data_1/total_counts`
- `raw_data_1/total_uncounted_counts`

## Potential differences from existing files

- `raw_data_1/total_uncounted_counts` - "uncounted count" may not be a valid concept in datastreaming world. But we can write it as a constant zero instead.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start by locating the filewriter and tracing how it receives data from the `_events` and other streams. Compare the existing metadata handling with the listed dynamic paths, including the per-frame calculation needed for `collection_time`; done means those paths are written during a run, with `total_uncounted_counts` handled according to the stated constant-zero possibility.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Bereich
backend, data-engineering, stream-processing
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.