dask / dask/distributed

Reconstructed Futures in deployed app always report pending status

Open
#3,890 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

It appears that Dask Futures always use _pending_ as the default value, which seems inappropriate.

In my use case, I have a Flask service with two routes:
1. `/submit` starts a long-running (several minutes-long) Dask job. This returns a Future key (plus some additional info relating to the request).
2. `/status/` that, given `key` will check the status of the job from step 1.

The result from step 1 is always a key with a status of _pending_ (since the job is just starting when my Flask service returns its JSON result). When I pass this key to the status request, I also always get a _pending_ status back -- even if the job has completed (verified by finding the result file that step 1 creates).

I have two concerns here:

1. If reconstruct a `Future` from a known valid key, the Dask client doesn't report the status as `finished` - this seems like a bug to me. If there's a proper way to do this, I'd love to find out. From the Python REPL, this does seem to work; from a deployed application, it doesn't.

2. If I construct a `Future` from an invalid key (eg, `fut = Future("this_key_doesnt_exist")`), then my Future's status shouldn't default to _pending_ but perhaps _invalid_. To me, _pending_ implies the job is running - especially since, from the REPL, a Future starts as _pending_ and moves to _finished_. Perhaps an interrmediate state explicitly indicating the job is _running_ or _executing_?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.