bpfman / bpfman/bpfman-operator

improve status events and/or messages

Open
#388 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
38
Forks
27
PR merge metrics
No merged PRs in 30d

Description

If we try to `kubectl apply` a `Kprobeprogram` with a kprobe for `accept4` syscalls like this:

```yaml
apiVersion: bpfman.io/v1alpha1
kind: KprobeProgram
metadata:
...
spec:
func_name: sys_accept4
...
```
it will fail. If we `kubectl describe` it, we will find a `Status` similar to this:

```yaml
Status:
Conditions:
Last Transition Time: 2025-02-27T13:28:39Z
Message: bpfProgramReconciliation failed on the following bpfProgram Objects: [my-kprobe-123456789]
Reason: ReconcileError
Status: True
Type: ReconcileError
Events:
```

Then, if we `kubectl describe BpfProgram my-kprobe-123456789`, we'll get the following status:

```yaml
Status:
Conditions:
Last Transition Time: 2025-02-27T13:28:45Z
Message: Failed to load bpfProgram
Reason: bpfmanNotLoaded
Status: True
Type: NotLoaded
Events:
```
Neither of which are really descriptive. If we then, go into the `bpfman-daemon` logs, more specifically, into the `bpfman-agent` logs, we'll find something like this:

```json
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"bpfman-agent enter: kprobe","Name":"my-kprobe"}
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"Calling KubeAPI to create BpfProgram","Name":"my-kprobe-123456789","Owner":"my-kprobe"}
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"bpfman-agent enter: kprobe","Name":"my-kprobe"}
{"level":"info","ts":"2025-02-27T13:28:39Z","logger":"kprobe","msg":"Calling bpfman to load eBPF Program on node","Name":"my-kprobe-123456789"}
{"level":"error","ts":"2025-02-27T13:28:45Z","logger":"kprobe","msg":"Failed to load eBPF Program","error":"failed to load bpfProgram via bpfman: rpc error: code = Aborted desc = `perf_event_open` failed","stacktrace":"github.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileBpfProgram\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:316\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).handleProgCreateOrUpdate\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:787\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileProgram\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:906\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileCommon\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:215\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*KprobeProgramReconciler).Reconcile\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/kprobe-program.go:190\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:249"}
{"level":"info","ts":"2025-02-27T13:28:45Z","logger":"kprobe","msg":"Calling KubeAPI to update BpfProgram condition","Name":"my-kprobe-123456789","condition":"NotLoaded"}
{"level":"error","ts":"2025-02-27T13:28:45Z","logger":"kprobe","msg":"Reconciling program failed","NameError":"json: unsupported type: func() string","ReconcileResult":"Updated","error":"failed to reconcile bpfman program: failed to load bpfProgram via bpfman: rpc error: code = Aborted desc = `perf_event_open` failed","stacktrace":"github.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*ReconcilerCommon[...]).reconcileCommon\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/common.go:217\ngithub.com/bpfman/bpfman-operator/controllers/bpfman-agent.(*KprobeProgramReconciler).Reconcile\n\t/usr/src/bpfman-operator/controllers/bpfman-agent/kprobe-program.go:190\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/usr/src/bpfman-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:249"}
```
Which is more descriptive, but it fails to convey the root cause, as the original error message from `bpfman` apparently gets truncated:

```json
"failed to load bpfProgram via bpfman: rpc error: code = Aborted desc = `perf_event_open` failed"
```

It should probably say something more like:

```
perf_event_open() failed: No such file or directory
```

To finally arrive at the root cause: there's no such symbol `sys_accept4`.

Perhaps we can start with checking if the error message is in fact truncated, and address that. Then, it would be a good idea to translate that into a more concrete message (e.g. "invalid symbol" or "symbol not found" or something similar) that would show up when describing the `Kprobeprogram` resource that is applied by the user initially.

So far I've only tried it on `amd64`, but it might also be the case for other platforms.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with controllers/bpfman-agent/common.go, especially reconcileBpfProgram and handleProgCreateOrUpdate, then compare the KprobeProgram flow in kprobe-program.go with the kubectl describe output and agent logs. Done means the load failure and its root cause are preserved in the relevant status or event, with the invalid-symbol case reported clearly; verify by reproducing the shown sys_accept4 apply.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.