pingcap / pingcap/ticdc

dispatchermanager: NewDispatcherManager may leak resources on init failure

Open
#4,606 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Problem

NewDispatcherManager may leak resources when initialization fails after sink.New(...) succeeds but before the manager is returned to the caller.

This is not introduced by the current storage sink changes. It appears to be a pre-existing rollback gap in the dispatcher manager initialization path.

Code path

downstreamadapter/dispatchermanager.NewDispatcherManager(...)

Current sequence:

  1. Create a private ctx/cancel for the manager.
  2. Create manager.sink via sink.New(...).
  3. Create sharedInfo.
  4. Register into HeartBeatCollector.
  5. Initialize table-trigger dispatcher / redo components.
  6. Start background goroutines and return the manager.

If an error happens in step 4 or step 5, the function returns early.

At that point:

  • the caller does not get a usable manager object, so normal manager.close() is never called;
  • the upper layer currently only calls HeartBeatCollector.RemoveDispatcherManager(...) on error;
  • manager.sink.Close(...), manager.sharedInfo.Close(), cancel(), and redo cleanup are not executed.
Why this matters

For storage sink specifically, sink.New(...) may already allocate resources before Run() starts, for example:

  • external storage clients / transports;
  • cloud storage sink statistics / metrics;
  • spool working directory and related local resources.

For some storage backends, ExternalStorage.Close() is not a no-op. For example, GCS closes underlying clients in Close().

So an init failure in NewDispatcherManager can leave resources alive even though the changefeed bootstrap fails.

Existing cleanup is not sufficient

The current upper-layer error handling in dispatcher_orchestrator.handleBootstrapRequest(...) only does:

  • log the error;
  • call HeartBeatCollector.RemoveDispatcherManager(changefeedID);
  • send the bootstrap error response.

That does not release the sink or other manager-owned resources.

Suggested fix

Add explicit init rollback for NewDispatcherManager(...):

  • call cancel() on failure after the private context is created;
  • if sink.New(...) succeeded, call manager.sink.Close(false);
  • if sharedInfo was created, call manager.sharedInfo.Close();
  • if redo components were partially initialized, close redo sink / redo meta as well;
  • remove heartbeat collector registration only if registration already succeeded.

The rollback should stay local to NewDispatcherManager(...), because that function owns the partial initialization state.

Scope

This issue is about dispatcher manager init rollback in general. It is adjacent to storage sink lifecycle work, but not caused by the storage sink PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at downstreamadapter/dispatchermanager.NewDispatcherManager(...) and trace each partial-initialization failure after the private context, sink, sharedInfo, heartbeat registration, and redo components are created. Verify that every owned resource is released on the relevant failure path, and that heartbeat removal occurs only after registration succeeds. Done means bootstrap failures leave no manager-owned resources running.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.