dask / dask/dask-gateway

Log provider API

Open
#77 12 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
148
Forks
93
PR merge metrics
No merged PRs in 30d

Description

Sometimes a user may want to look at logs of completed workers/clusters. Right now all log handling is backend specific - users need to be familiar with the cluster backend and the particularities of how logs are handled for that backend. For example, YARN logs are stored to HDFS and can be accessed with the `yarn` cli tool.

It may be useful for `dask-gateway` to provide a `LogProvider` class that different log backends could implement. This might look like:

```python
class LogProvider(LoggingConfigurable):
def get_logs_for_cluster(self, cluster_name, cluster_state):
"""Get the logs for a completed cluster

Parameters
----------
cluster_name : str
The cluster name.
cluster_state : dict
Any backend-specific information (e.g. application id, pod name, ...)

Returns
-------
logs : dict[str, str]
A mapping from job id to logs for that job.
"""

def get_logs_for_worker(self, cluster_name, cluster_state, worker_name, worker_state):
"""get the logs for a completed worker"""
```

I'd prefer dask-gateway doesn't manage the storage of these logs (although we could if needed), rather it should be an abstraction around accessing the logs wherever they're being held by some other service/convention.

Possible implementations for our cluster backends:

- YARN: this is hard as YARN has no java api for this, but we could hack something up
- Jobqueue: filesystem backed, logs could be stored in `~/dask-gateway-logs` per user, or in some directory managed directly by dask-gateway?
- Kubernetes: I'm not sure? There's lots of possible services people might use for logs here. [Stackdriver](https://cloud.google.com/logging/) perhaps?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.