pingcap / pingcap/tidb

backup/restore: new pipeline implementation

Open
#49,719 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

(Description WIP.)

### Target
An uniformed, observable pipeline implementation.

### The interface

Some principles:

- Split the pipeline runtime and the pipeline implementation: generally the implementation should care less things about the runtime.
- The runtime spawns new goroutines, manifesting the metadata provided by the worker implementation.
- The worker should have a small interface so the caller will be easy to implement.
- The worker should be composable. Prefer combine pipeline workers by theirselves and provide some interfaces for the runtime to inspect the inner structure of workers instead of let the runtime to manage the shape of the pipeline.

A draft version: “Context is all you need.”

```go
type Context[Out any] interface {
context.Context

Emit(out Out) error
EmitErr(err error)
Finish()
}

type Worker[In, Out any] interface {
MainLoop(ctx Context[Out], input <-chan In)
}
```

“But why there is an extra `input`?”
“`Context` cannot be selected. Channel is the first-class citizen in Go.”

(Still WIP. Current bigger problem: how can we embed `Spawn` to `Context`? It should be a generic function but we cannot let the interface provide a generic function.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.