apache / apache/shenyu

[BUG] Failback retry exhaustion leaves a permanently stuck entry that silently swallows all future failed registrations for the same key

Open
#6,545 1 comment 0 reactions 0 assignees View on GitHub
client: register priority: high type: bug
Dominant language
Java
Stars
8.8k
Forks
3.1k
Avg merge
7d 1h
Merged PRs (30d)
85

Description

## Description
`FailureRegistryTask` is constructed with `retryCount = 18`, so `retryLimit = (18 < 0) = false`. After 18 consecutive failures, `tickCount` reaches 19; the guard `!retryLimit && tickCount > retryCount` is true, so `run()` logs "task over" and returns **without** calling `remove(key)` and **without** rescheduling. The `Holder` for that key stays in `concurrentHashMap` forever. `addToFail` then short-circuits on the stuck entry: `if (Objects.nonNull(oldObj)) { return; }` — no new `FailureRegistryTask` is ever scheduled again for that key.

## Location
```
shenyu-common/.../timer/AbstractRetryTask.java:107-110 (exhaustion branch — returns without cleanup)
shenyu-register-client-api/.../retry/FailureRegistryTask.java:54-58 (doRetry — only removes on success)
shenyu-register-client-api/.../FailbackRegistryRepository.java:175-184 (addToFail — early-returns when key present)
```

## Impact
After ~3 minutes (18 x 10s) of sustained failure the admin becomes reachable again, but the client never re-registers that URI/metadata because every subsequent failure for the same key hits the stuck entry. Also a memory leak (`concurrentHashMap` grows with dead entries).

## Suggested fix
In the `AbstractRetryTask.run()` exhaustion branch, call a cleanup hook (e.g. `onGiveUp(key)` overridden by `FailureRegistryTask` to call `registerRepository.remove(key)`) before returning, so future failures can schedule a fresh task. Alternatively, have `doRetry` remove the entry in a `finally` and re-add on next failure.

## Related existing issue(s)
None

_Identified during the 2026-08-02 audit; full list in [`docs/issue-candidates-2026-08-02.md`](docs/issue-candidates-2026-08-02.md)._

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the exhaustion branch in shenyu-common/.../timer/AbstractRetryTask.java alongside doRetry in shenyu-register-client-api/.../retry/FailureRegistryTask.java and addToFail in shenyu-register-client-api/.../FailbackRegistryRepository.java. Verify the retry entry is cleaned up after exhaustion, a later failure can schedule a fresh task for the same key, and dead entries do not remain in concurrentHashMap.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.