kernelci / kernelci/kernelci-project

Lava callback latency problem

Open
#446 1 comment 0 reactions 0 assignees View on GitHub
techdebt
Dominant language
Python
Stars
14
Forks
32
Avg merge
1d 1h
Merged PRs (30d)
3

Description

Right now callback is fastapi, which is async. But callback endpoint is slow, as it need to upload artifacts to kernelci storage.
Even i did threads, it seems still slow. We have grafana monitoring that does plain HTTP request to default / endpoint, and it is
reaching 2-4 seconds latency when we have lot of LAVA submissions.
We need to investigate this, and probably rewrite way how callback works fundamentally.
**It might be also totally different issue (for example docker-proxy or nginx), because k8s deployment (production) handling this well, but staging (based on docker containers) is slow.**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the FastAPI callback path, including artifact uploads to KernelCI storage, and compare the staging Docker deployment with the production Kubernetes deployment. Use the Grafana request to the default / endpoint as the latency signal, then investigate whether the bottleneck is the callback, docker-proxy, or nginx. Done means the latency cause is identified and a concrete callback or deployment fix is validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, fastapi, grafana, kubernetes, nginx, python
Domain
backend, infrastructure, observability, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.