NVIDIA / NVIDIA/nvcf

[FEA][observability] Support end-to-end request tracing and adopter-defined correlation headers

Open
#1,034 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-triage
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

After a successful NVCF invocation, the response includes an NVCF-REQID
header. We would like guidance and support for using this request identifier
to achieve end-to-end, per-request observability across the complete invocation
path.

We also need a way for NVCF adopters to provide their own correlation identifier
or custom metadata header when invoking a function.

Use cases

1. End-to-end request observability and tracing

Given the NVCF-REQID returned in the response, how can an operator:

  • Trace the request across the NVCF invocation plane, routing components, and
    the deployed function?
  • Find all logs and OpenTelemetry spans associated with the request?
  • Correlate the request with function ID, cluster ID, organization ID, and
    downstream calls?
  • Export this trace information to an adopter-managed observability backend?
  • Troubleshoot failed or timed-out requests when a response containing
    NVCF-REQID might not be returned?

It would be helpful to document whether NVCF propagates W3C traceparent and
tracestate headers and how NVCF-REQID maps to an OpenTelemetry trace or span.

2. Adopter-defined correlation header

Can an adopter send a custom correlation header with an invocation, for example:

X-Correlation-ID: adopter-request-123

If supported:

  • Is the header forwarded through the invocation plane to the function container?
  • Is it included in structured logs or OpenTelemetry span attributes?
  • Can logs and traces be searched or filtered using its value?
  • Is the header returned in the response?
  • Which header names and value formats are allowed?
  • Are there size, security, redaction, or retention restrictions?

If arbitrary custom headers cannot be propagated safely, an allowlisted,
first-class header such as NVCF-CORRELATION-ID would satisfy the use case.

The correlation value should be available as a searchable log field or trace
attribute. It should not be used as a Prometheus metric label because request-level
values have unbounded cardinality.

Expected outcome

Please provide either:

  1. Documentation describing the currently supported end-to-end tracing and
    custom-header behavior; or
  2. A feature that:
    • propagates W3C trace context across NVCF services;
    • correlates NVCF-REQID with logs and traces;
    • supports a validated adopter-defined correlation ID;
    • forwards that correlation ID to the deployed function;
    • exposes it as a searchable structured-log field and OpenTelemetry span
      attribute; and
    • documents security, size, and redaction rules.

Related work

This appears related to #16, Bring Your Own Observability support, but this issue
focuses specifically on per-invocation correlation and adopter-provided request
identifiers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing NVCF-REQID handling across the invocation plane, routing components, and deployed function, then compare it with related issue #16. Done means documenting current tracing and header behavior or implementing the requested propagation, searchable correlation fields, and security, size, and redaction rules.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus
Domain
api, cloud, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.