Pipeline refactoring proposal

While thinking of ways of making KernelCI more flexible for future requirements, I identified some design elements as the sources of the hardships we've been finding so far during development and as potential flexibility limitations:

• The current pipeline stages are separated but aren't really working in an independent way: even if they're independent stages, the way they are meant to work right now is as linear pipeline stages: ie. the output of one stage feeds into the next and some stages expect an input from the stage right before.
• The scheduler stage conflates scheduling logic and job generation/running, so it's not trivial to test-run a job generation without involving the whole scheduler machinery and configuration.

I think this pipeline design was probably done as a first sketch and then we've been stacking patches on top of it, but I think the KernelCI API design is meant to be used in a different way by the pipeline and other clients. As it often happens during rushed designs, once we got a hammer we started using it for hammering nails, but also for screws and to punch in holes in the walls, instead of finding ways to get us a screwdriver and a drill as well.

So this is a proposal for change to overcome these limitations and make KernelCI both easier to develop on and easier to adapt to future use cases.

Current test flow

As I understand it, the current flow for running a test looks something like this (the numbers in parentheses show the event order):

Proposal

1. Introduce an additional "Runner" stage that generates jobs and runs it. That is, move the second half of the current "Scheduler" to a separate stage and leave the "Scheduler" to handle exclusively the logic of which jobs to run.
2. Make a better use of events to communicate clients together. If we introduce a new type of event, such as run (this is free to do afaik) we can use it to kickstart processes in individual stages selectively from any source. This event may contain all the necessary data to describe the event and for the target stage to perform the task.
3. Introduce a new (maybe optional) "Job Dispatcher" stage that can take high-level job descriptions and then trigger the appropriate stages to get a result. A job description could be simply a json definition to "run a test in a specific platform on a specific kernel version and setup", and the dispatcher would decompose that into specific stage runs to fetch the kernel code, build it and run the test.

An example scenario would look like this (note: this isn't a sequence diagram, the order of API interactions between components doesn't matter):

So, the current test flow could still work just the same, the sequence would be:

1. "Trigger" detects a repo change, or it receives a run event addressed to it. Then it does its usual tasks, submits the checkout node and notifies the "tarball" stage by sending a run event.
2. "Tarball" starts when it receives a run event addressed to it, it does the usual tasks and submits the node update
3. The "Scheduler" works as usual, receiving node events, with the difference that instead of running the jobs itself, it decides which jobs to run based on the pipeline configuration and notifies the "Runner" stage to run them, passing all the necessary information about the job in the run event.
4. The "Runner" generates the jobs and run them as it receives them via run events.

Additionally, this decoupling of functionalities allows other use cases that we can't do at the moment:

• A client can trigger an individual stage to have KernelCI perform an operation on demand: build a kernel remotely and upload it (running a standalone kbuild), run a test with a custom kernel build
• External tools can plug into the API at any stage
• Higher-level processes and logic can be built on top of these primitive stages

No source files, tests, or entry points are named. Start by mapping the current stage and event flow from the issue's current-flow description, then review how Scheduler combines scheduling with job generation and running. Done would be an agreed, actionable design for Runner, event-based stage communication, and the optional Job Dispatcher, including scoped implementation work.