ISISComputingGroup / ISISComputingGroup/DataStreaming
`filewriter`: file archiving
- Ngôn ngữ chính
- Không có dữ liệu ngôn ngữ
- Star
- 0
- Fork
- 0
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
The file-writing infrastructure must be able to trigger file-archiving workflows to get the file from wherever it is generated, to places where it can be read by downstream consumers and stored according to the ISIS data policy.
This includes:
- Generating file checksums
* See [comment from @ChrisM-S on ADR to drop .raw files](https://github.com/ISISComputingGroup/ibex_developers_manual/pull/236#issuecomment-5192402543) for further details about how file checksumming is currently done
- Marking files with read-only attribute [for existing windows ISIS archive]
- Moving files from the location where they were generated to the archive
* In the immediate term this is assumed to be the existing ISIS archive, however the chosen approach should be flexible enough to cope with longer-term architectural direction in this area (for example see EPAC 'data pipeline' slides).
* Including sufficiently robust processes to "catch up" and recover gracefully later if the archive is offline or unreachable at the time a file is generated, or the copy onto the archive fails.
- Any other steps which are required to go from some `.nxs` bytes on disk, to a file that a user can download from ISIS data gateway, that mantid can read from its expected locations, and gets archived to SCD long-term tape store.
This functionality does not necessarily have to be in the filewriter itself; it may be better to live *outside* the filewriter process, as long as the filewriter provides adequate 'hooks' for triggering archiving (e.g. `wrdn`)
## Questions
- Do files, or a copy, go _via_ the instrument (NDX) data area at all? Which downstream processes depend on getting data from there - IDAAAAAS, Mantid, AutoReduction, ...?
## Potential differences from existing files
- File checksums are currently written into a Windows alternate file stream in the `.raw` file. This almost certainly is not the best approach for a linux-based filewriter. Do any downstream consumers other than the archival process currently use these checksums embedded in alternate file streams?
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Start by mapping the filewriter and its possible archiving hooks, including `wrdn`, against the existing ISIS archive and the EPAC data-pipeline direction. Investigate whether data passes through the instrument NDX area and which consumers depend on it, including IDAAAAAS, Mantid, and AutoReduction. Done means an agreed, recoverable path from generated `.nxs` bytes through checksum and archive handling to the data gateway and long-term tape store.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Lĩnh vực
- data, infrastructure
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 25/100