globus / globus/globus-compute

parsl provider error messages are lost

Open
#679 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
162
Forks
53
Avg merge
15h 29m
Merged PRs (30d)
26

Description

**Describe the bug**
This is based on a report in the #help slack channel

When the slurm provider fails to scale out, the code that is supposed to report that to the user fails in potentially several ways:

1) This seems to be static type error in the exception handling code for scale_out failing, when constructing a more specific exception - interchange indeed has no `config`.

```
Submission of command to scale_out failed
2022-01-27 14:33:58.605 funcx_endpoint.strategies.simple:43 [ERROR] Caught error in strategize : 'Interchange' object has no attribute 'config'
Traceback (most recent call last):
File "/work2/04372/ejonas/anaconda/envs/s2s/lib/python3.9/site-packages/funcx_endpoint/strategies/simple.py", line 41, in strategize
self._strategize(*args, **kwargs)
File "/work2/04372/ejonas/anaconda/envs/s2s/lib/python3.9/site-packages/funcx_endpoint/strategies/simple.py", line 143, in _strategize
self.interchange.scale_out(excess_blocks)
File "/work2/04372/ejonas/anaconda/envs/s2s/lib/python3.9/site-packages/funcx_endpoint/executors/high_throughput/interchange.py", line 1151, in scale_out
self.config.provider.label,
AttributeError: 'Interchange' object has no attribute 'config'
```

2. The parsl layer logs an error to eg `parsl.providers.slurm` but the endpoint admin was unable to find the relevant log message - maybe it should appear around the same place as the above report? The relevant parsl log line is:

```
logger.error("Retcode:%s STDOUT:%s STDERR:%s", retcode, stdout.strip(), stderr.strip())
```

**To Reproduce**
Get endpoint to try to scale out with a broken provider/provider configuration

**Expected behavior**
The errors coming from parsl.providers should lead the user towards fixing the problem (in the example user's case, a quota exhaustion reported by `sbatch`) rather than being hidden

**Environment**
slurm
other component versions unknown

Contributor guide

Open the contributing guide

Research direction

Start by tracing scale-out error handling in funcx_endpoint/strategies/simple.py and executors/high_throughput/interchange.py, especially the reported access to interchange.config. Then inspect the parsl.providers.slurm error logging shown in the issue and reproduce scale-out with a broken provider configuration. Done means the provider error, including useful stdout or stderr such as quota failures, reaches the endpoint administrator without being hidden or replaced by another exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.