apache / apache/openwhisk

Scheduler "stops" after a while

Open
#5,485 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

## Environment details:

* AWS EKS 1.29
* Tag 7863e27 for scheduler, controller, invoker

## Steps to reproduce the issue:

1. Deploy using scheduler etc...
2. deploy some functions (cron based etc...

## Provide the expected results and outputs:

```
The system works normally
```

## Provide the actual results and outputs:

After a while ( about an hour). The system becomes unstable.

In the controller log:

```
[2024-05-21T13:00:09.119Z] [ERROR] [#tid_kDcLqa1uLqzR7GhLLdejqhGeS3mbeHQo] [] Failed to recreate queue for dataspace/ncarpark/carpark@0.0.1, no scheduler endpoint available
```

Also seeing this in the controller log:

```
[2024-05-21T13:00:06.174Z] [WARN] [#tid_kDcLqa1uLqzR7GhLLdejqhGeS3mbeHQo] [] The whisk/queue/dataspace/dataspace/carpark/carpark/leader is deleted from ETCD, but there are still unhandled activations for this action, try to create a new queue
```

In the scheduler log I am seeing this:

```
[2024-05-21T13:00:09.876Z] [WARN] [#tid_sid_unknown] [EtcdWorker] a lease is expired while registering an initial data whisk/queue/dataspace/dataspace/carpark/carpark/leader, reissue it: io.grpc.StatusRuntimeException: NOT_FOUND: etcdserver: requested lease not found
```

And also:

```
[2024-05-21T13:00:10.195Z] [WARN] [#tid_sid_unknown] [EtcdWorker] a lease is expired while registering an initial data whisk/scheduler/0, reissue it: io.grpc.StatusRuntimeException: NOT_FOUND: etcdserver: requested lease not found
```

Not sure where to look for this issue

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.