flyteorg / flyteorg/flyte

[BUG] Task not running (but queued) when no resource meeting requirements can be found

Open
#6,417 1 comment 0 reactions 0 assignees View on GitHub
bug untriaged
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 14h
Merged PRs (30d)
120

Description

### Describe the bug

I am running a local Kubernetes cluster and noticed that my tasks show up as "Running" in the UI, but after a veery long time they were still not finished. Actually, they did not run at all.
The task that I wanted to run had a resource allocator requesting 32gb of RAM, which was in preparation of deploying it to a cluster on the cloud. The local version would just be using a subset of the data and therefore not require that much RAM.

After removing the resource requirement the task ran very quickly.

Since there are not really logs in the UI except for running tasks, a label called "Running" is misleading here as the task is actually not running at all, but is queued (and will ultimately never run as the requirement can not be met).

### Expected behavior

The task being stuck at "Queued", fail or something else than "Running".

### Additional context to reproduce

1. Create a task that requests more resources than what is available
2. Try to run the task

Example task:
```python
@task(requests=Resources(mem="32Gi"))
def get_my_data() -> pl.DataFrame:
"""Read data from redshift."""
query = "select * from my_table"

# Read the data from Redshift
df = pl.read_database(query, connection=settings.redshift_connection)

return df
```

### Screenshots

Clicking on the task it actually shows "queued", but the label says "Running"

![Image](https://github.com/user-attachments/assets/cb9c1ebb-231b-47ad-b68d-7cccb6049a14)

### Are you sure this issue hasn't been raised already?

- [x] Yes

### Have you read the Code of Conduct?

- [x] Yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.