globus / globus/globus-compute

kubernetes provider does not notice missing pods removed by anything other than itself

Open
#705 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
162
Forks
53
Avg merge
15h 29m
Merged PRs (30d)
26

Description

**Describe the bug**
If I kill a worker pod, then
when I launch a subsequent task, hoping that the system will start up a new worker pod, instead my task progresses as far as `Task is pending due to waiting-for-nodes` and a new worker pod is not launched. This looks like its because the fork of the kubernetes provider in funcx does not check kubernetes for status, and continues to claim that worker pod exists -- this was fixed in the fork of kubernetes provider in parsl in early 2021 - see https://github.com/Parsl/parsl/pull/1740.
I thought I'd already opened a funcx github issue on this but I can't find it.

Restarting the endpoint clears away the list of disappeared-pods.

This 2nd issue is a little bit disguised by scaling: once I have blocked the first missing container with sufficent hung tasks, the end point scales out a new pod to take on any excess work - which then succeeds to execute any new work. So a user experiencing this who accepts that "often funcx doesn't run very well, i should just keep retrying and not report a problem" will trigger that effect without reporting a problem

**To Reproduce**
Delete a worker pod

**Expected behavior**
Something more like the parsl fork of the kubernetes provider, https://github.com/Parsl/parsl/pull/1740

**Environment**
my kubernetes dev environment, main branches as of 2022-02-28

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.