backup/restore: new pipeline implementation
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
(Description WIP.)
### Target
An uniformed, observable pipeline implementation.
### The interface
Some principles:
- Split the pipeline runtime and the pipeline implementation: generally the implementation should care less things about the runtime.
- The runtime spawns new goroutines, manifesting the metadata provided by the worker implementation.
- The worker should have a small interface so the caller will be easy to implement.
- The worker should be composable. Prefer combine pipeline workers by theirselves and provide some interfaces for the runtime to inspect the inner structure of workers instead of let the runtime to manage the shape of the pipeline.
A draft version: “Context is all you need.”
```go
type Context[Out any] interface {
context.Context
Emit(out Out) error
EmitErr(err error)
Finish()
}
type Worker[In, Out any] interface {
MainLoop(ctx Context[Out], input <-chan In)
}
```
“But why there is an extra `input`?”
“`Context` cannot be selected. Channel is the first-class citizen in Go.”
(Still WIP. Current bigger problem: how can we embed `Spawn` to `Context`? It should be a generic function but we cannot let the interface provide a generic function.)
Contributor guide
Assessment
This issue has not been assessed yet.