Block Relval workflows expecting as an input datasets missing from disk
- Dominant language
- Python
- Stars
- 48
- Forks
- 111
- PR merge metrics
- No merged PRs in 30d
Description
**Impact of the new feature**
MSTransferor or Workflow submission
**Is your feature request related to a problem? Please describe.**
This is a followup on the following Jira ticket: https://its.cern.ch/jira/browse/CMSTRANSF-486
Long Story short:
* We have a limited lifetime for the RleVal output datasets - max 3 months to stay on disk and no Tape copy
* There are known use cases, where a newly submitted Relval may require as an input a dataset as described above
* If such a RelVal is submitted upon the dataset lifetime on disk, no warning is shot during submission and the workflow gets stuck, which requires later investigation and leads to additional operational burden.
**Describe the solution you'd like**
As discussed in the Jira ticker, several options are available to tackle this behavior. I am repeating them here:
* Implementing extra checks about the origin of the input and the delay between the two possibly connected workflows (the one producing the input and the one producing the output) during the assignment step. (I am not sure if this is not already in place).
* Resolving the "out of sync" situation between DBS and Rucio and let GWQ do its job during the data discovery process (already in place).
* Implementing extra checks about data content presence in MSTransferror through some extra and potentially heavy calls to Rucio during the staging step (not implemented).
Since we do not yet own the assignor module, the first option is not solvable though the WMcore system - it would require changes in Unified. But we may (as @amaltaro suggested in the jira ticket) lift that check a step earlier during workflow submission and swap the check for the input presence to be made not trough DBS but trough Rucio directly. That would leave us with the following two feasible choices of solving the problem:
* Change the checks for input data integrity from DBS to Rucio at submission time, and if not autmatically blocking the workflow at least issue a waning to the submitter
* Make changes to MSTransferor module to check for the data integrity thee and automatically block those workflows during the staging step.
**Describe alternatives you've considered**
All alternatives listed in the previous bullet
**Additional context**
None
Contributor guide
Assessment
This issue has not been assessed yet.