pingcap / pingcap/tiproxy

Potential concurrency bugs in ConfigManager

Open
#532 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
73
Forks
41
Avg merge
21h 3m
Merged PRs (30d)
21

Description

Development Task

ConfigManager notifies other modules with the config change by a channel. There are some bugs:

  1. When a module starts, it must receive an initial config and a channel as its parameters, such as someModule.Init(cfgMgr.GetConfig(), cfgMgr.WatchConfig()). However, between calling GetConfig() and WatchConfig(), there may be a config change, and thus someModule may miss this config change.
  2. ConfigManager notifies all the modules by writing to the channels when config changes. If one module blocks (maybe forever or temporarily), it hangs. If the module blocks forever, such as deadlock or panic, the config change will block forever. But if you add a timeout on writing the channel, the module may miss the config change if it just blocks temporarily, such as busy handling other things.
  3. There's no way to unsubscribe from ConfigManager after calling WatchConfig(). If a module wants to quit, ConfigManager will still try to notify it.

For the 1st problem, one workaround is calling WatchConfig() after GetConfig().
For the 2nd problem, one workaround is to set the channel size to a big enough size and also set a timeout on writing the channel.
For the 3rd problem, one workaround is to add another function to unsubscribe.

If we deprecate the WatchConfig() way and always use GetConfig(), all problems will be solved.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating ConfigManager and its GetConfig and WatchConfig call sites. Review how initialization, notification blocking, and watcher lifetime are handled; done requires an agreed design that addresses missed updates, blocked modules, and unsubscription, along with corresponding tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.