globus / globus/globus-compute
parsl provider error messages are lost
- Dominant language
- Python
- Stars
- 162
- Forks
- 53
- Avg merge
- 15h 29m
- Merged PRs (30d)
- 26
Description
**Describe the bug**
This is based on a report in the #help slack channel
When the slurm provider fails to scale out, the code that is supposed to report that to the user fails in potentially several ways:
1) This seems to be static type error in the exception handling code for scale_out failing, when constructing a more specific exception - interchange indeed has no `config`.
```
Submission of command to scale_out failed
2022-01-27 14:33:58.605 funcx_endpoint.strategies.simple:43 [ERROR] Caught error in strategize : 'Interchange' object has no attribute 'config'
Traceback (most recent call last):
File "/work2/04372/ejonas/anaconda/envs/s2s/lib/python3.9/site-packages/funcx_endpoint/strategies/simple.py", line 41, in strategize
self._strategize(*args, **kwargs)
File "/work2/04372/ejonas/anaconda/envs/s2s/lib/python3.9/site-packages/funcx_endpoint/strategies/simple.py", line 143, in _strategize
self.interchange.scale_out(excess_blocks)
File "/work2/04372/ejonas/anaconda/envs/s2s/lib/python3.9/site-packages/funcx_endpoint/executors/high_throughput/interchange.py", line 1151, in scale_out
self.config.provider.label,
AttributeError: 'Interchange' object has no attribute 'config'
```
2. The parsl layer logs an error to eg `parsl.providers.slurm` but the endpoint admin was unable to find the relevant log message - maybe it should appear around the same place as the above report? The relevant parsl log line is:
```
logger.error("Retcode:%s STDOUT:%s STDERR:%s", retcode, stdout.strip(), stderr.strip())
```
**To Reproduce**
Get endpoint to try to scale out with a broken provider/provider configuration
**Expected behavior**
The errors coming from parsl.providers should lead the user towards fixing the problem (in the example user's case, a quota exhaustion reported by `sbatch`) rather than being hidden
**Environment**
slurm
other component versions unknown
Contributor guide
Research direction
Start by tracing scale-out error handling in funcx_endpoint/strategies/simple.py and executors/high_throughput/interchange.py, especially the reported access to interchange.config. Then inspect the parsl.providers.slurm error logging shown in the issue and reproduce scale-out with a broken provider configuration. Done means the provider error, including useful stdout or stderr such as quota failures, reaches the endpoint administrator without being hidden or replaced by another exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100