bpfman / bpfman/bpfman-operator
improve status events and/or messages
- Dominant language
- Go
- Stars
- 38
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
If we try to `kubectl apply` a `Kprobeprogram` with a kprobe for `accept4` syscalls like this:
```yaml
apiVersion: bpfman.io/v1alpha1
kind: KprobeProgram
metadata:
...
spec:
func_name: sys_accept4
...
```
it will fail. If we `kubectl describe` it, we will find a `Status` similar to this:
```yaml
Status:
Conditions:
Last Transition Time: 2025-02-27T13:28:39Z
Message: bpfProgramReconciliation failed on the following bpfProgram Objects: [my-kprobe-123456789]
Reason: ReconcileError
Status: True
Type: ReconcileError
Events:
```
Then, if we `kubectl describe BpfProgram my-kprobe-123456789`, we'll get the following status:
```yaml
Status:
Conditions:
Last Transition Time: 2025-02-27T13:28:45Z
Message: Failed to load bpfProgram
Reason: bpfmanNotLoaded
Status: True
Type: NotLoaded
Events:
```
Neither of which are really descriptive. If we then, go into the `bpfman-daemon` logs, more specifically, into the `bpfman-agent` logs, we'll find something like this:
```json
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"bpfman-agent enter: kprobe","Name":"my-kprobe"}
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"Calling KubeAPI to create BpfProgram","Name":"my-kprobe-123456789","Owner":"my-kprobe"}
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"bpfman-agent enter: kprobe","Name":"my-kprobe"}
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"Calling bpfman to load eBPF Program on node","Name":"my-kprobe-123456789"}
{"level":"error","ts":"2025-02-27T13:28:45Z","logger":"kprobe","msg":"Failed to load eBPF Program","error":"failed to load bpfProgram via bpfman: rpc error: code = Aborted desc = `perf_event_open` failed","stacktrace":"github.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileBpfProgram\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:316\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).handleProgCreateOrUpdate\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:787\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileProgram\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:906\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileCommon\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:215\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*KprobeProgramReconciler).Reconcile\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/kprobe-program.go:190\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:249"}
{"level":"info","ts":"2025-02-27T13:28:45Z","logger":"kprobe","msg":"Calling KubeAPI to update BpfProgram condition","Name":"my-kprobe-123456789","condition":"NotLoaded"}
{"level":"error","ts":"2025-02-27T13:28:45Z","logger":"kprobe","msg":"Reconciling program failed","NameError":"json: unsupported type: func() string","ReconcileResult":"Updated","error":"failed to reconcile bpfman program: failed to load bpfProgram via bpfman: rpc error: code = Aborted desc = `perf_event_open` failed","stacktrace":"github.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileCommon\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:217\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*KprobeProgramReconciler).Reconcile\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/kprobe-program.go:190\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:249"}
```
Which is more descriptive, but it fails to convey the root cause, as the original error message from `bpfman` apparently gets truncated:
```json
"failed to load bpfProgram via bpfman: rpc error: code = Aborted desc = `perf_event_open` failed"
```
It should probably say something more like:
```
perf_event_open() failed: No such file or directory
```
To finally arrive at the root cause: there's no such symbol `sys_accept4`.
Perhaps we can start with checking if the error message is in fact truncated, and address that. Then, it would be a good idea to translate that into a more concrete message (e.g. "invalid symbol" or "symbol not found" or something similar) that would show up when describing the `Kprobeprogram` resource that is applied by the user initially.
So far I've only tried it on `amd64`, but it might also be the case for other platforms.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with controllers/bpfman-agent/common.go, especially reconcileBpfProgram and handleProgCreateOrUpdate, then compare the KprobeProgram flow in kprobe-program.go with the kubectl describe output and agent logs. Done means the load failure and its root cause are preserved in the relevant status or event, with the invalid-symbol case reported clearly; verify by reproducing the shown sys_accept4 apply.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100