hard ordering constraints in scripts in mirrors.
- Dominant language
- C
- Stars
- 4
- Forks
- 1
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 1
Description
It is a clear limitation of the entire method of mirroring that order of operations are not strictly maintained. Files are posted, mostly in the order the changes are created, then the events are queued. There are subscribers that pick from the queue and in the case of HPC mirroring, the queue is shared among 40 subscribers... so any event can be in the queue for any of the subscribers.
Since different subscribers can pick up different events, and the subscribers are asynchronous to each other, the events can be executed on the subscriber side out of order with respect to the source. This "out of order" ness is intrinsic to the copying algorithm. Getting rid of out of order will force sequential operations which is expected to have a large performance impact... both in sychnronization primitives among the subscribers, and in reduction of operations that can proceed in parallel.
Note that, between the posting layer and the subscribing layer, there is the *winnow* layer. The winnow layer delays copies for 30 seconds, and squashes multiple changes to the same file so that only a single update is produced (if the file is re-written 10 times in 30 seconds, a post will be published for that file 30 seconds after the last write.) This is to de-noise the copies... so extremely transient files are not copied.
So we have a script. it does:
```
rm X
ln -sf y X
```
but the mirroring logic can re-arrange things so that the ln arrives and is executed before the rm. so instead of getting a new contents of X linked to y, X is just gone.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the posting, winnow, and subscribing layers described in the issue, including the shared queue and asynchronous subscribers. Define how ordering constraints for script operations such as rm X followed by ln -sf y X should be detected and preserved without removing all parallelism. Done means those dependent operations execute in source order while the mirroring behavior remains validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100