ReactionMechanismGenerator / ReactionMechanismGenerator/ARC

Socket time out error when error handling

Open
#394 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Bug Type: Feature
Dominant language
Python
Stars
51
Forks
25
Avg merge
4d 5h
Merged PRs (30d)
15

Description

Recently, C3DDB server is suffering from multiple issues: file system issues and queue software issues, which helps us identify more cases ARC needs to consider. For example, today, I (manually) and ARC couldn't check the job status by running squeue -u xiaorui for several. It will return an error like

squeue: error: slurm_receive_msg: Socket timed out on send/recv operation, 
slurm_load_jobs error: Socket timed out on send/recv operation

Currently, I find a case where this can be problematic. When ARC submits a new job and cannot read the queue info, It will regard the job as finished. However, when it reads the outputs, it will find that the job has never started. ARC will try another launch, which, of course, fails again. Then the job will be terminated and marked as having an error. I guess there can be other potential risks. It is worthy to have ARC handling this kind of situation.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named in the issue. Start by locating ARC's job submission and queue-status handling, then reproduce the failure with squeue -u xiaorui and its socket-timeout output. Done means an unreadable queue is not treated as a finished job or causes an erroneous relaunch and termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.