crab status wrongly reports a DAG bootstrap failure
- Dominant language
- Python
- Stars
- 18
- Forks
- 38
- PR merge metrics
- No merged PRs in 30d
Description
see https://hypernews.cern.ch/HyperNews/CMS/get/computing-tools/1852.html
I think we do not want to fix this now, since @emaszs is rewriting the status, but I log so that he can check that the new implementation is using proper care in handling what looks like a race condition in current implementation.
here[1] is the relevant part of client log, I have removed all the noise from WMCore/Proxy stuff, relevant line is:
```
ERROR 2016-07-06 08:39:22,004: Error retrieving task status: The task failed to bootstrap on the Grid scheduler <128.142.136.249:4080?addrs=128.142.136.249-4080+[--1]-4080&noUDP&sock=2779_bdc7_3>.
```
while 1 minute later a second crab status reports
```
DEBUG 2016-07-06 08:40:23,436: {'status': 'SUBMITTED',...
```
Seems there is a condition where crab status does not correctly interpret the DAG status.
The DAG appears OK, with no restarts (i.e. no failures):
```
belforte@vocms0109/~> condor_q 9118943
-- Schedd: crab3-2@vocms0109.cern.ch : <128.142.136.249:4080?...
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
9118943.0 cms1229 7/6 08:39 1+08:35:16 R 10 0.1 dag_bootstrap_startup.sh RunJobs.dag
1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended
belforte@vocms0109/~> condor_q -l 9118943|grep -i start
JobCurrentStartDate = 1467787165
NumRestarts = 0
StageInStart = 1467787161
NumJobStarts = 1
Cmd = "dag_bootstrap_startup.sh"
JobStartDate = 1467787165
belforte@vocms0109/~>
```
that start date correspond to:
```
belforte@vocms0109/~> date -d @1467787165
Wed Jul 6 08:39:25 CEST 2016
```
i.e. the DAG bootstrap apparently started 3 seconds after the client reported error.
[1]
```
DEBUG 2016-07-06 08:39:15,862: Executing command: 'status'
DEBUG 2016-07-06 08:39:15,862: Found CRAB cache file /afs/cern.ch/user/p/piedra/.crab3
DEBUG 2016-07-06 08:39:15,869: Checking credentials
[...]
DEBUG 2016-07-06 08:39:19,411: Myproxy is valid: 1408473
DEBUG 2016-07-06 08:39:19,411: Command use: status
DEBUG 2016-07-06 08:39:19,411: Options use: ['crabdir/crab_DoubleMuon_Run2016B-PromptReco-v2/']
DEBUG 2016-07-06 08:39:19,700: CRABClient version: 3.3.1606
DEBUG 2016-07-06 08:39:19,700: Instance is prod
DEBUG 2016-07-06 08:39:19,700: Server base url is cmsweb.cern.ch
DEBUG 2016-07-06 08:39:19,700: Command url /crabserver/prod/workflow
DEBUG 2016-07-06 08:39:19,700: Looking up detailed status of task 160706_063815:piedra_crab_DoubleMuon_Run2016B-PromptReco-v2
DEBUG 2016-07-06 08:39:22,004: {'status': 'UNKNOWN', 'username': 'piedra', 'totalJobdefs': 0, 'jobsPerStatus': {}, 'taskWarningMsg': [], 'schedd': 'crab3-2@vocms0109.cern.ch', 'statusFailureMsg': 'T
INFO 2016-07-06 08:39:22,004: CRAB project directory: /afs/cern.ch/work/p/piedra/public/CMSSW_projects/CMSSW_8_0_5/src/LatinoTrees/AnalysisStep/test/crab/crabdir/crab_DoubleMuon_Run2016B-PromptReco
INFO 2016-07-06 08:39:22,004: Task name: 160706_063815:piedra_crab_DoubleMuon_Run2016B-PromptReco-v2
INFO 2016-07-06 08:39:22,004: Grid scheduler: crab3-2@vocms0109.cern.ch
INFO 2016-07-06 08:39:22,004: Task status: UNKNOWN
INFO 2016-07-06 08:39:22,004: Dashboard monitoring URL: http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=crab&refresh=0&table=Jobs&p=1&records=25&activemenu=2&status=&site=&tid=1
ERROR 2016-07-06 08:39:22,004: Error retrieving task status: The task failed to bootstrap on the Grid scheduler <128.142.136.249:4080?addrs=128.142.136.249-4080+[--1]-4080&noUDP&sock=2779_bdc7_3>. Please send an e-mail to hn-cms-computing-tools@cern.ch. Hold reason: None
INFO 2016-07-06 08:39:22,007: Log file is /afs/cern.ch/work/p/piedra/public/CMSSW_projects/CMSSW_8_0_5/src/LatinoTrees/AnalysisStep/test/crab/crabdir/crab_DoubleMuon_Run2016B-PromptReco-v2/crab.log
DEBUG 2016-07-06 08:40:15,407: Executing command: 'status'
DEBUG 2016-07-06 08:40:15,408: Found CRAB cache file /afs/cern.ch/user/p/piedra/.crab3
DEBUG 2016-07-06 08:40:15,421: Checking credentials
[...]
DEBUG 2016-07-06 08:40:18,422: Myproxy is valid: 1408414
DEBUG 2016-07-06 08:40:18,422: Command use: status
DEBUG 2016-07-06 08:40:18,422: Options use: ['crabdir/crab_DoubleMuon_Run2016B-PromptReco-v2/']
DEBUG 2016-07-06 08:40:18,659: CRABClient version: 3.3.1606
DEBUG 2016-07-06 08:40:18,659: Instance is prod
DEBUG 2016-07-06 08:40:18,659: Server base url is cmsweb.cern.ch
DEBUG 2016-07-06 08:40:18,659: Command url /crabserver/prod/workflow
DEBUG 2016-07-06 08:40:18,659: Looking up detailed status of task 160706_063815:piedra_crab_DoubleMuon_Run2016B-PromptReco-v2
DEBUG 2016-07-06 08:40:23,436: {'status': 'SUBMITTED', 'username': 'piedra', 'jobsPerStatus': {'unsubmitted': 1887}, 'jobs': {'643': {'State': 'unsubmitted'}, '592': {'State': 'unsubmitted'}, '344':
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.