[FEA][observability] Support end-to-end request tracing and adopter-defined correlation headers
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Description
After a successful NVCF invocation, the response includes an NVCF-REQID
header. We would like guidance and support for using this request identifier
to achieve end-to-end, per-request observability across the complete invocation
path.
We also need a way for NVCF adopters to provide their own correlation identifier
or custom metadata header when invoking a function.
Use cases
1. End-to-end request observability and tracing
Given the NVCF-REQID returned in the response, how can an operator:
- Trace the request across the NVCF invocation plane, routing components, and
the deployed function? - Find all logs and OpenTelemetry spans associated with the request?
- Correlate the request with function ID, cluster ID, organization ID, and
downstream calls? - Export this trace information to an adopter-managed observability backend?
- Troubleshoot failed or timed-out requests when a response containing
NVCF-REQIDmight not be returned?
It would be helpful to document whether NVCF propagates W3C traceparent and
tracestate headers and how NVCF-REQID maps to an OpenTelemetry trace or span.
2. Adopter-defined correlation header
Can an adopter send a custom correlation header with an invocation, for example:
X-Correlation-ID: adopter-request-123
If supported:
- Is the header forwarded through the invocation plane to the function container?
- Is it included in structured logs or OpenTelemetry span attributes?
- Can logs and traces be searched or filtered using its value?
- Is the header returned in the response?
- Which header names and value formats are allowed?
- Are there size, security, redaction, or retention restrictions?
If arbitrary custom headers cannot be propagated safely, an allowlisted,
first-class header such as NVCF-CORRELATION-ID would satisfy the use case.
The correlation value should be available as a searchable log field or trace
attribute. It should not be used as a Prometheus metric label because request-level
values have unbounded cardinality.
Expected outcome
Please provide either:
- Documentation describing the currently supported end-to-end tracing and
custom-header behavior; or - A feature that:
- propagates W3C trace context across NVCF services;
- correlates
NVCF-REQIDwith logs and traces; - supports a validated adopter-defined correlation ID;
- forwards that correlation ID to the deployed function;
- exposes it as a searchable structured-log field and OpenTelemetry span
attribute; and - documents security, size, and redaction rules.
Related work
This appears related to #16, Bring Your Own Observability support, but this issue
focuses specifically on per-invocation correlation and adopter-provided request
identifiers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing NVCF-REQID handling across the invocation plane, routing components, and deployed function, then compare it with related issue #16. Done means documenting current tracing and header behavior or implementing the requested propagation, searchable correlation fields, and security, size, and redaction rules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus
- Domain
- api, cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100