dmwm / dmwm/CRABServer

Investigate moving schedd HTC work files to a separate disk

Open
#7,166 1 comment 0 reactions 0 assignees View on GitHub
Area: HTCondor Priority: Low Status: Available Status: On Hold Type: Enhancement
Dominant language
Python
Stars
21
Forks
42
Avg merge
1d 2h
Merged PRs (30d)
9

Description

Our schedulers need a lot of I/O operations since both HTCondor scheduler daemon and CRAB 's various scripts use disk files for all bookkeeping and message passing.
We are not I/O limited now. But we have suffered of I/O limitation in the past, before we moved SPOOL areas (/data disk) to High-I/O CEPH volumes.

Before we run into issues again, it would make sense to separate CRAB and HTCondor I/O traffic using separate CEPH volume for scheduler internal log and queue. Need to check with condor experts exactly which files needs to be moved beside the "obvious" `/data/srv/glidecondor/condor_local/job_history/`, `/data/srv/glidecondor/condor_local/log` , `/data/srv/glidecondor/condor_local/spool/history*`, `/data/srv/glidecondor/condor_local/spool/*queue*`. Not that we need to keep those (which are HTC work ones) separate from `/data/srv/glidecondor/condor_local/spool/1234` ones which contain the spool area of each task in which both CRAB and Scheduler write.
It is not obvious to Stefano that this will be enough, since schedd may still block on writing to individual task spool disk, but Brian thought it could help, and Brian is always right !

Maybe not change config of all schedulers w/o a clear need, but try to do this on a test one and measure I/O performance improvements (if any) to be ready in case of need.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by consulting Condor experts and inventory the scheduler work files, including /data/srv/glidecondor/condor_local/job_history/, log, spool/history*, and spool/*queue*. Test the separation on one scheduler rather than changing all configurations, then measure I/O performance. Done means the required HTC files are identified and the test documents whether separating them improves performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.