dmwm / dmwm/CRABClient

crab status wrongly reports a DAG bootstrap failure

Open
#4,632 0 comments 0 reactions 1 assignee Claimed by @emaszs View on GitHub
bug Low Priority
Dominant language
Python
Stars
18
Forks
38
PR merge metrics
No merged PRs in 30d

Description

see https://hypernews.cern.ch/HyperNews/CMS/get/computing-tools/1852.html

I think we do not want to fix this now, since @emaszs is rewriting the status, but I log so that he can check that the new implementation is using proper care in handling what looks like a race condition in current implementation.

here[1] is the relevant part of client log, I have removed all the noise from WMCore/Proxy stuff, relevant line is:

```
ERROR 2016-07-06 08:39:22,004: Error retrieving task status: The task failed to bootstrap on the Grid scheduler <128.142.136.249:4080?addrs=128.142.136.249-4080+[--1]-4080&noUDP&sock=2779_bdc7_3>.
```

while 1 minute later a second crab status reports

```
DEBUG 2016-07-06 08:40:23,436: {'status': 'SUBMITTED',...
```

Seems there is a condition where crab status does not correctly interpret the DAG status.
The DAG appears OK, with no restarts (i.e. no failures):

```
belforte@vocms0109/~> condor_q 9118943

-- Schedd: crab3-2@vocms0109.cern.ch : <128.142.136.249:4080?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
9118943.0 cms1229 7/6 08:39 1+08:35:16 R 10 0.1 dag_bootstrap_startup.sh RunJobs.dag

1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended
belforte@vocms0109/~> condor_q -l 9118943|grep -i start
JobCurrentStartDate = 1467787165
NumRestarts = 0
StageInStart = 1467787161
NumJobStarts = 1
Cmd = "dag_bootstrap_startup.sh"
JobStartDate = 1467787165
belforte@vocms0109/~>
```

that start date correspond to:

```
belforte@vocms0109/~> date -d @1467787165
Wed Jul 6 08:39:25 CEST 2016
```

i.e. the DAG bootstrap apparently started 3 seconds after the client reported error.

[1]

```
DEBUG 2016-07-06 08:39:15,862: Executing command: 'status'
DEBUG 2016-07-06 08:39:15,862: Found CRAB cache file /afs/cern.ch/user/p/piedra/.crab3
DEBUG 2016-07-06 08:39:15,869: Checking credentials
[...]
DEBUG 2016-07-06 08:39:19,411: Myproxy is valid: 1408473
DEBUG 2016-07-06 08:39:19,411: Command use: status
DEBUG 2016-07-06 08:39:19,411: Options use: ['crabdir/crab_DoubleMuon_Run2016B-PromptReco-v2/']
DEBUG 2016-07-06 08:39:19,700: CRABClient version: 3.3.1606
DEBUG 2016-07-06 08:39:19,700: Instance is prod
DEBUG 2016-07-06 08:39:19,700: Server base url is cmsweb.cern.ch
DEBUG 2016-07-06 08:39:19,700: Command url /crabserver/prod/workflow
DEBUG 2016-07-06 08:39:19,700: Looking up detailed status of task 160706_063815:piedra_crab_DoubleMuon_Run2016B-PromptReco-v2
DEBUG 2016-07-06 08:39:22,004: {'status': 'UNKNOWN', 'username': 'piedra', 'totalJobdefs': 0, 'jobsPerStatus': {}, 'taskWarningMsg': [], 'schedd': 'crab3-2@vocms0109.cern.ch', 'statusFailureMsg': 'T
INFO 2016-07-06 08:39:22,004: CRAB project directory: /afs/cern.ch/work/p/piedra/public/CMSSW_projects/CMSSW_8_0_5/src/LatinoTrees/AnalysisStep/test/crab/crabdir/crab_DoubleMuon_Run2016B-PromptReco
INFO 2016-07-06 08:39:22,004: Task name: 160706_063815:piedra_crab_DoubleMuon_Run2016B-PromptReco-v2
INFO 2016-07-06 08:39:22,004: Grid scheduler: crab3-2@vocms0109.cern.ch
INFO 2016-07-06 08:39:22,004: Task status: UNKNOWN
INFO 2016-07-06 08:39:22,004: Dashboard monitoring URL: http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=crab&refresh=0&table=Jobs&p=1&records=25&activemenu=2&status=&site=&tid=1
ERROR 2016-07-06 08:39:22,004: Error retrieving task status: The task failed to bootstrap on the Grid scheduler <128.142.136.249:4080?addrs=128.142.136.249-4080+[--1]-4080&noUDP&sock=2779_bdc7_3>. Please send an e-mail to hn-cms-computing-tools@cern.ch. Hold reason: None
INFO 2016-07-06 08:39:22,007: Log file is /afs/cern.ch/work/p/piedra/public/CMSSW_projects/CMSSW_8_0_5/src/LatinoTrees/AnalysisStep/test/crab/crabdir/crab_DoubleMuon_Run2016B-PromptReco-v2/crab.log
DEBUG 2016-07-06 08:40:15,407: Executing command: 'status'
DEBUG 2016-07-06 08:40:15,408: Found CRAB cache file /afs/cern.ch/user/p/piedra/.crab3
DEBUG 2016-07-06 08:40:15,421: Checking credentials
[...]
DEBUG 2016-07-06 08:40:18,422: Myproxy is valid: 1408414
DEBUG 2016-07-06 08:40:18,422: Command use: status
DEBUG 2016-07-06 08:40:18,422: Options use: ['crabdir/crab_DoubleMuon_Run2016B-PromptReco-v2/']
DEBUG 2016-07-06 08:40:18,659: CRABClient version: 3.3.1606
DEBUG 2016-07-06 08:40:18,659: Instance is prod
DEBUG 2016-07-06 08:40:18,659: Server base url is cmsweb.cern.ch
DEBUG 2016-07-06 08:40:18,659: Command url /crabserver/prod/workflow
DEBUG 2016-07-06 08:40:18,659: Looking up detailed status of task 160706_063815:piedra_crab_DoubleMuon_Run2016B-PromptReco-v2
DEBUG 2016-07-06 08:40:23,436: {'status': 'SUBMITTED', 'username': 'piedra', 'jobsPerStatus': {'unsubmitted': 1887}, 'jobs': {'643': {'State': 'unsubmitted'}, '592': {'State': 'unsubmitted'}, '344':
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.