elastic / elastic/cloudbeat

Enhance the channel buffer to enhance the concurrency (currently working without buffered channels)

Open
#739 0 comments 0 reactions 0 assignees View on GitHub
cloudbeat Team:Cloud Security technical debt Vulnerability Management
Dominant language
Go
Stars
58
Forks
55
Avg merge
9h 9m
Merged PRs (30d)
424

Description

The whole vulnerability flow is divided using channels.
Every component works on an input channel and outputs to the next channel.
The components run as `goroutines` in parallel (currently with a scale of 1, preferably scale them according to available cores).
These channels are non-buffered as of right now, which means that this flow happens in a serial manner.
We would like to consider and change that behavior in order to make sure to parallelize some of the work.
This can be done easily by converting the channels to buffered channels and adding a loop for every component that we want to scale likewise:
```go
func (f *VulnerabilityWorker) Run(ctx context.Context) {
// TODO: Handle deletion of snapshots
f.log.Info("Starting VulnerabilityWorker.work")
for {
select {
case <-ctx.Done():
f.log.Info("VulnerabilityWorker.work context canceled")
return
default:
f.wg.Add(1)
go func() {
defer f.wg.Done()
f.fetcher.FetchInstances(ctx)
f.log.Info("VulnerabilityWorker.work FetchInstances finished")
}()
*FOR LOOP TO SCALE*
f.wg.Add(1)
go func() {
defer f.wg.Done()
f.replicator.SnapshotInstance(ctx, f.fetcher.GetChan())
f.log.Info("VulnerabilityWorker.work SnapshotInstance finished")
}()
f.wg.Add(1)
go func() {
defer f.wg.Done()
f.verifier.VerifySnapshot(ctx, f.replicator.GetChan())
f.log.Info("VulnerabilityWorker.work VerifySnapshot finished")
}()
f.wg.Add(1)
go func() {
defer f.wg.Done()
f.evaluator.EvaluateSnapshot(ctx, f.verifier.GetChan())
f.log.Info("VulnerabilityWorker.work EvaluateSnapshot finished")
}()
*FOR LOOP TO SCALE END*
f.log.Info("VulnerabilityWorker.work waiting on workers")
f.wg.Wait()
f.log.Info("VulnerabilityWorker.work finished waiting on workers")
return
}
}
}
```
This ticket depends on the progress and implementation of the following ticket:
- https://github.com/elastic/cloudbeat/issues/691

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.