Import errors on workers should be visible in output
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
This issue was originally opened on 29 Feb 2020 as https://github.com/dask/dask-cloudprovider/issues/72. I'm moving it to this repo per https://github.com/dask/dask-cloudprovider/issues/72#issuecomment-599613817. I have not confirmed that the issue is still present; I apologise if it has been fixed in the interim.
The code below fails because a required module is not present on the workers. However, the error output does not contain any mention of the import error. It would be very helpful for debugging if this remote import error could be captured and form part of the local stderr stream.
```python
from dask.distributed import Client
from dask_cloudprovider import FargateCluster
import dask.bag
from this_module_is_present_locally_but_not_on_worker import example_task
cluster = FargateCluster(n_workers=1)
client = Client(cluster)
print(dask.bag.from_sequence(range(1)).map(example_task).compute())
```
where `this_module_is_present_locally_but_not_on_worker.py` contains
```python
print("Importing module: this_module_is_present_locally_but_not_on_worker")
def example_task(i):
print("In function: example_task")
return i + 1
```
```
dask==2.11.0
dask-cloudprovider==0.1.1
```
The above code yields the following output (stdout + stderr):
```python-traceback
Importing module: this_module_is_present_locally_but_not_on_worker
/Users/dan/.pyenv/versions/3.7.0/lib/python3.7/contextlib.py:119: UserWarning: Creating your cluster is taking a surprisingly long time. This is likely due to pending resources on AWS. Hang tight!
next(self.gen)
Traceback (most recent call last):
File "dask_cloudprovider_minimal_example.py", line 10, in
print(dask.bag.from_sequence(range(1)).map(example_task).compute())
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/dask/base.py", line 166, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/dask/base.py", line 437, in compute
results = schedule(dsk, keys, **kwargs)
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/distributed/client.py", line 2595, in get
results = self.gather(packed, asynchronous=asynchronous, direct=direct)
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/distributed/client.py", line 1893, in gather
asynchronous=asynchronous,
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/distributed/client.py", line 780, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync
raise exc.with_traceback(tb)
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/distributed/utils.py", line 332, in f
result[0] = yield future
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/Users/dan/tmp/virtualenvs/elaenia/lib/python3.7/site-packages/distributed/client.py", line 1752, in _gather
raise exception.with_traceback(traceback)
distributed.scheduler.KilledWorker: ("('example_task-5cfd48a7dc92a2f12f6991607e897d49', 0)", )
distributed.client - ERROR - Failed to reconnect to scheduler after 10.00 seconds, closing client
_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError()>
concurrent.futures._base.CancelledError
```
The example works as expected if I add the line
```python
client.upload_file("this_module_is_present_locally_but_not_on_worker.py")
```
Contributor guide
Assessment
This issue has not been assessed yet.