hashicorp / hashicorp/nomad

volume watcher should back off if plugin unavailable

Open
#16,728 0 comments 0 reactions 0 assignees View on GitHub
theme/storage type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

While testing out some other issues, I discovered that if a plugin is completely removed but there are allocations with claims, the volumewatcher gets into a tight error loop. This only seems to happen if there are terminal allocations with claims have not yet been reaped. Everything recovers as expected if the plugin is re-registered.

> 2023-03-30T11:39:09.016-0400 [ERROR] nomad.volumes_watcher: error releasing volume claims: namespace=default volume_id=volume1
> error=
> | 1 error occurred:
> | \t* no such plugin: "hostpath-plugin0"
> |

(Also these are ugly multi-errors when they don't need to be. Would be nice to fix that while we're in there.)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the volume watcher and reproduce the tight error loop with a removed plugin, terminal allocations, and unreaped claims. Verify the behavior when the plugin is re-registered, then make completion mean that unavailable plugins are retried with backoff and the unnecessary multi-error output is cleaned up.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.