dask / dask/dask-cloudprovider

Cloud logs observability improvements

Open
#470 6 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

`dask-cloudprovider` integration might be tough unless the cloud is configured correctly. Often, the developer might see that the initial start of the cluster simply hung, and Dask waits infinitely for the scheduler to be started:
```
Creating scheduler instance
dask-5424b11a-scheduler
Internal IP: 10.128.0.220
External IP: 35.184.140.159
Waiting for scheduler to run at 35.184.140.159:8786
```
Usually, the problem is observed in cloud-init logs or docker container std output that can fetched if you connect to the instance directly via SSH and cat cloud-init logs:
```
cat /var/log/cloud-init-output.log
```
Or fetches the Docker container stdout:
```
docker logs
```

I propose to output these logs into a `distributed` logger.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.