kernelci / kernelci/kernelci-pipeline

Pipeline services assume event payload to be Dictionary

Open
#481 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
11
Forks
40
Avg merge
2d 13h
Merged PRs (30d)
14

Description

Pipeline services such as monitor and scheduler are based on PubSub events published on `node` channel.
While parsing event data, the services assume it to be a `Dict` and post-process data accordingly.

Previously, event payload data was [Dict](https://github.com/kernelci/kernelci-api/commit/7cd15778d6194e07c74f8e46e90d2ba80a5dc936) and hence the the pipeline services were implemented like that. But now it has changed to be [Any](https://github.com/kernelci/kernelci-api/commit/5764b7abadd1a627f8ccb07bd2ad6bf4a3e0ce9e) type. Thus, the issue with data post-processing in the pipeline has surfaced.

e.g. take a look at the below code section of monitor service:
```
event = self._api.receive_event(sub_id)
obj = event.data
dt = datetime.datetime.fromisoformat(event['time'])
commit = (obj['data']['kernel_revision']['commit'][:12]
if 'kernel_revision' in obj['data']
else str(None))
```

If somehow, `str` data is sent to `node` channel, it will end up in error.

Monitor service failure:
```
today at 12:11:55 File "/home/kernelci/pipeline/base.py", line 69, in run
today at 12:11:55 status = self._run(context)
today at 12:11:55 ^^^^^^^^^^^^^^^^^^
today at 12:11:55 File "/home/kernelci/./pipeline/monitor.py", line 64, in _run
today at 12:11:55 if 'kernel_revision' in obj['data']
today at 12:11:55 ~~~^^^^^^^^
today at 12:11:55TypeError: string indices must be integers, not 'str'
today at 12:11:55
today at 12:11:55Container stopped
```

Also, scheduler service with `shell` runtime:
```
today at 12:11:5503/18/2024 06:41:55 AM UTC [ERROR] Traceback (most recent call last):
today at 12:11:55 File "/home/kernelci/pipeline/base.py", line 69, in run
today at 12:11:55 status = self._run(context)
today at 12:11:55 ^^^^^^^^^^^^^^^^^^
today at 12:11:55 File "/home/kernelci/./pipeline/scheduler.py", line 189, in _run
today at 12:11:55 for job, runtime, platform, rules in self._sched.get_schedule(event):
today at 12:11:55 File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 44, in get_schedule
today at 12:11:55 for config in self.get_configs(event, channel):
today at 12:11:55 File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 39, in get_configs
today at 12:11:55 if sched_event.items() <= event.items():
today at 12:11:55 ^^^^^^^^^^^
today at 12:11:55AttributeError: 'str' object has no attribute 'items'
```

Scheduler with `docker`:
```
today at 12:11:5503/18/2024 06:41:55 AM UTC [ERROR] Traceback (most recent call last):
today at 12:11:55 File "/home/kernelci/pipeline/base.py", line 69, in run
today at 12:11:55 status = self._run(context)
today at 12:11:55 ^^^^^^^^^^^^^^^^^^
today at 12:11:55 File "/home/kernelci/./pipeline/scheduler.py", line 189, in _run
today at 12:11:55 for job, runtime, platform, rules in self._sched.get_schedule(event):
today at 12:11:55 File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 44, in get_schedule
today at 12:11:55 for config in self.get_configs(event, channel):
today at 12:11:55 File "/usr/local/lib/python3.11/site-packages/kernelci/scheduler.py", line 39, in get_configs
today at 12:11:55 if sched_event.items() <= event.items():
today at 12:11:55 ^^^^^^^^^^^
today at 12:11:55AttributeError: 'str' object has no attribute 'items'
today at 12:11:55
today at 12:11:55Traceback (most recent call last):
today at 12:11:55 File "/home/kernelci/./pipeline/scheduler.py", line 218, in
today at 12:11:55 status = opts.command(configs, opts)
today at 12:11:55 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 12:11:55 File "/home/kernelci/./pipeline/scheduler.py", line 211, in __call__
today at 12:11:55 return Scheduler(configs, args).run()
today at 12:11:55 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
today at 12:11:55 File "/home/kernelci/pipeline/base.py", line 76, in run
today at 12:11:55 self._stop(context)
today at 12:11:55 File "/home/kernelci/./pipeline/scheduler.py", line 73, in _stop
today at 12:11:55 self._cleanup_paths()
today at 12:11:55 File "/home/kernelci/./pipeline/scheduler.py", line 58, in _cleanup_paths
today at 12:11:55 job_tmp_dirs = {
today at 12:11:55 ^
today at 12:11:55 File "/home/kernelci/./pipeline/scheduler.py", line 62, in
today at 12:11:55 if job.poll() is None
today at 12:11:55 ^^^^^^^^
today at 12:11:55AttributeError: 'Container' object has no attribute 'poll'
today at 12:13:49Container stopped
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pipeline/monitor.py and pipeline/scheduler.py, tracing how event.data reaches monitor processing and Scheduler.get_schedule(). Reproduce a node-channel event whose payload is a string, then inspect the scheduler paths for shell and Docker runtimes, including the cleanup failure in _cleanup_paths(). Done means these services handle the reported payload and runtime cases without the shown exceptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.