dask / dask/dask-cloudprovider

Task._update_task crashes with "not enough values to unpack" when describe_tasks returns empty tasks

Open Beginner friendly
#483 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

`Task._update_task` in `dask_cloudprovider/aws/ecs.py` unpacks the ECS `describe_tasks` response into exactly one element with no guard:

[self.task] = (
await ecs.describe_tasks(cluster=self.cluster_arn, tasks=[self.task_arn])
)["tasks"]

ECS can return an empty `tasks` list (with the task under `failures` as `MISSING`) during the RunTask -> DescribeTasks read-after-write consistency window — the ~1s right after a task is created. When that happens the unpack raises `ValueError: not enough values to unpack (expected 1, got 0)`, which `distributed` rewraps as `RuntimeError: Cluster failed to start`, killing `FargateCluster` startup.

This is the same defect as [#33](https://github.com/dask/dask-cloudprovider/issues/33) / [#36](https://github.com/dask/dask-cloudprovider/pull/36) ("add task error handling"), which added a guard to the sibling `run_task` unpack in `start()` but left `_update_task` unguarded. Worse, `_update_task` is called from `start()` *after* its 60s retry loop, so the error is terminal rather than retried.

**Minimal Complete Verifiable Example**:

```python
# The real trigger is a timing race: a FargateCluster whose scheduler task is
# polled in the brief window where ECS DescribeTasks still reports it MISSING.
# from dask_cloudprovider.aws import FargateCluster
# cluster = FargateCluster(...) # intermittently fails on startup
#
# Deterministic reduction — describe_tasks returns this documented shape
# during the consistency window, and _update_task does the unguarded unpack:
response = {"tasks": [], "failures": [{"reason": "MISSING"}]}
[task] = response["tasks"] # ValueError: not enough values to unpack (expected 1, got 0)
```

**Anything else we need to know?**:

Observed against a real run: ECS `RunTask` for the scheduler succeeded (one task returned, `failures: []`, status `PROVISIONING`); the very next `DescribeTasks` ~1s later returned `tasks: []` / `failures: [{reason: MISSING}]`.

Suggested fix — mirror #36 inside `_update_task`'s loop so an empty `tasks` (transient `MISSING`) retries instead of crashing:

resp = await ecs.describe_tasks(cluster=self.cluster_arn, tasks=[self.task_arn])
if not resp["tasks"]:
await asyncio.sleep(wait_duration)
continue
[self.task] = resp["tasks"]

The bug is present on the latest release and on `main`.

**Environment**:

- Dask version: dask-cloudprovider 2025.9.0; dask / distributed 2025.5.1
- Python version: 3.11
- Operating System: Linux (AWS Fargate container)
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Research direction

Start in dask_cloudprovider/aws/ecs.py at Task._update_task, then compare its describe_tasks handling with the guarded unpack in start() and the behavior described in #36. Reproduce the empty tasks response from the issue and verify that the transient MISSING result is retried rather than terminating FargateCluster startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.