Log provider API
- Dominant language
- Python
- Stars
- 148
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Sometimes a user may want to look at logs of completed workers/clusters. Right now all log handling is backend specific - users need to be familiar with the cluster backend and the particularities of how logs are handled for that backend. For example, YARN logs are stored to HDFS and can be accessed with the `yarn` cli tool.
It may be useful for `dask-gateway` to provide a `LogProvider` class that different log backends could implement. This might look like:
```python
class LogProvider(LoggingConfigurable):
def get_logs_for_cluster(self, cluster_name, cluster_state):
"""Get the logs for a completed cluster
Parameters
----------
cluster_name : str
The cluster name.
cluster_state : dict
Any backend-specific information (e.g. application id, pod name, ...)
Returns
-------
logs : dict[str, str]
A mapping from job id to logs for that job.
"""
def get_logs_for_worker(self, cluster_name, cluster_state, worker_name, worker_state):
"""get the logs for a completed worker"""
```
I'd prefer dask-gateway doesn't manage the storage of these logs (although we could if needed), rather it should be an abstraction around accessing the logs wherever they're being held by some other service/convention.
Possible implementations for our cluster backends:
- YARN: this is hard as YARN has no java api for this, but we could hack something up
- Jobqueue: filesystem backed, logs could be stored in `~/dask-gateway-logs` per user, or in some directory managed directly by dask-gateway?
- Kubernetes: I'm not sure? There's lots of possible services people might use for logs here. [Stackdriver](https://cloud.google.com/logging/) perhaps?
Contributor guide
Assessment
This issue has not been assessed yet.