Investigate moving schedd HTC work files to a separate disk
- Dominant language
- Python
- Stars
- 21
- Forks
- 42
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 9
Description
Our schedulers need a lot of I/O operations since both HTCondor scheduler daemon and CRAB 's various scripts use disk files for all bookkeeping and message passing.
We are not I/O limited now. But we have suffered of I/O limitation in the past, before we moved SPOOL areas (/data disk) to High-I/O CEPH volumes.
Before we run into issues again, it would make sense to separate CRAB and HTCondor I/O traffic using separate CEPH volume for scheduler internal log and queue. Need to check with condor experts exactly which files needs to be moved beside the "obvious" `/data/srv/glidecondor/condor_local/job_history/`, `/data/srv/glidecondor/condor_local/log` , `/data/srv/glidecondor/condor_local/spool/history*`, `/data/srv/glidecondor/condor_local/spool/*queue*`. Not that we need to keep those (which are HTC work ones) separate from `/data/srv/glidecondor/condor_local/spool/1234` ones which contain the spool area of each task in which both CRAB and Scheduler write.
It is not obvious to Stefano that this will be enough, since schedd may still block on writing to individual task spool disk, but Brian thought it could help, and Brian is always right !
Maybe not change config of all schedulers w/o a clear need, but try to do this on a test one and measure I/O performance improvements (if any) to be ready in case of need.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by consulting Condor experts and inventory the scheduler work files, including /data/srv/glidecondor/condor_local/job_history/, log, spool/history*, and spool/*queue*. Test the separation on one scheduler rather than changing all configurations, then measure I/O performance. Done means the required HTC files are identified and the test documents whether separating them improves performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100