[QA] Test request: NVCA OTel Collector Extension for Event Ledger (#809)

Description

QA validation request for the work delivered under epic #809 — NVCA OTel Collector Extension for Event Ledger.

This epic extends the FnDs (Event Ledger) ingest path so that per-instance lifecycle/failure detail is captured for both container and helm NVCF functions (and tasks), via a two-lane model, with client-side correlation across the lanes on icms_request_id.

What shipped

• NVCA — stamps icms_request_id on Pods/ICMSRequest events and maps it to the generic resource_id on the ICMS lane (#1509).
• byoo-otel-collector — cluster-wide receiver shipping two lanes (Pod + ICMSRequest) to POST /v3/ledger/k8s-events.
• event-ledger (FnDs) — generic, optional resource_id context field for non-Pod resources (#1117) and a generic attribute_key/attribute_value filter on GET /events (#1119). Also dotted/bulk CloudEvent support (#1142).

Two-lane model to validate

1. Pod lane — kubelet/scheduler Pod events. Context key: cluster_id + deployment_id + gpu_specification_id + instance_id. icms_request_id present as a payload attribute (details.attributes).
2. Resource (ICMS) lane — NVCA ICMSRequest.* events. Context key: cluster_id + resource_id (where resource_id = icms_request_id).
3. Correlation — join Pod ↔ ICMS lanes on icms_request_id (container + helm).

Endpoints under test (FnDs v3)

• GET /v3/ledger/namespace/{namespace}/events?cluster_id=…&resource_id=… (resource lane)
• GET /v3/ledger/namespace/{namespace}/events?cluster_id=…&deployment_id=…&gpu_specification_id=…&instance_id=… (pod lane)
• Correlation: append &attribute_key=icms_request_id&attribute_value=<icms_request_id> to filter within a context.
• namespace = function-version-id.

Environment

• Validated by the dev team end-to-end on a live staging cluster (nvcf-dgxc-k8s-oci-ord-ct3, org nvcf-eng-staging) by deploying both container and helm functions.
• Pinned versions: event-ledger 0.15.0, nvca/nvca-operator 3.5.2, byoo-otel-collector 0.157.0-nv-0.2.1.
• A Postman collection for the v3 APIs is available and can be shared for hands-on testing.

Definition of Done (QA)

• Container function: Pod lane returns full pod lifecycle events; every event carries icms_request_id.
• Helm function: Pod lane events captured from per-request sr-<uuid> namespace (no regression vs container).
• Resource lane: ICMSRequest.* events retrievable via resource_id for both function types.
• Correlation: Pod ↔ ICMS lanes join correctly on icms_request_id (container + helm).
• attribute_key/attribute_value filter: both-required contract enforced (supplying only one → 400 Bad Request).
• Failure scenarios: failure/termination events surface on the appropriate lane.
• Auth: v3 read endpoints reject non-read-scoped/invalid tokens (403), accept valid read-scoped SSA tokens.
• No regression to the existing container Pod → FnDs path.

Resources

Epic: #809 (sub-issues #810–#817, all completed)

Delivering PRs (all merged):

NVCA

• #938 — stamp FnDs ledger annotations on ICMSRequest Events + per-instance Event correlator (#810, #811)
• #1509 — stamp icms_request_id/resource_id for ledger correlation (#809)

byoo-otel-collector

• #940 — cluster-wide k8sobjects receiver + Pod lane for Event Ledger (#812, #813)

event-ledger (FnDs)

• #1117 — generic resource_id context field for non-Pod resources (#815)
• #1119 — generic attribute filter on GET /events (#816)
• #1142 — support dotted instance IDs and bulk CloudEvent writes

Reference: FnDs v3 API — src/control-plane-services/event-ledger/README.md

Start with the FnDs v3 API reference at src/control-plane-services/event-ledger/README.md and use the available Postman collection against the staging cluster. Test the resource and Pod lane endpoints for container and Helm functions, including correlation, filtering, failure events, and token checks. Done means every listed QA checkbox passes without regression to the existing container Pod → FnDs path.