NVIDIA / NVIDIA/nvcf

feat(request-trace-uploader): add Kratos Bulk Upload backend

Open
#1,443 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Why

Internal customers export request records to Kratos; external customers cannot reach Kratos at all and need the generic object store. The uploader must serve both from one image by changing only the backend setting.

Kratos Bulk Upload is not an S3 endpoint. It is a job API that internally vends short-lived STS credentials and PUTs to S3, then reports terminal status asynchronously. It is reached through the internal Go SDK at golang.nvidia.com/kratos/kratos-go-sdk, which is NVIDIA-internal and unlicensed.

Scope

  • Implement the Kratos backend against the existing backend interface from #1047. No new interface, no second uploader implementation.

  • Place the implementation behind a build tag so the core and the generic object-store backend compile without the internal SDK. A build without the tag must report that the Kratos backend is not compiled in, rather than failing to build.

  • Submit with UploadContext(..., kratosWaitForCompletion: false) and poll the exported GetUploadStatus separately, so submission and confirmation stay independent steps.

  • Own the terminal-status map rather than using the SDK's checkUploadStatus. That helper only breaks on SUCCESS or FAILED BULK UPLOAD INGESTION, so in storage-only mode it polls to its 1800s ceiling and reports a timeout on a job that actually succeeded at SUCCESS TEMP BUCKET UPLOAD.

  • Model all seven Kratos statuses, including the PENDING TEMP BUCKET UPLOAD, PENDING BULK UPLOAD INGESTION, and PROCESSING intermediates that the shell uploader ignores.

  • Declare capabilities honestly. Kratos V1 mints the upload ID server-side, so resubmit safety is false and indeterminate submissions must resolve to retain-and-quarantine rather than resubmit. V2 accepts a client-supplied upload ID; the SDK currently selects V2 only for the china profiles.

  • Classify Kratos errors into the retryable, terminal, and ambiguous classes and into segment or backend scope.

  • Supply the uploader's own HTTP client through NewBulkUploadKratosOptionsWithHttpClient so Retry-After handling and per-call latency instrumentation are possible.

  • Bound the SDK's internal 100-attempt S3 retry configuration with the configured per-attempt timeout via context.

  • Configuration: namespace, environment, ingestion toggle, database and table, mode, ACE ID, S3 endpoint, region, status poll interval and deadline. Environment accepts only global-prod, china-production, and china-staging; reject the shell uploader's production and staging vocabulary explicitly at startup.

  • Credentials come from the read-only mount by key name, never from values in configuration.

  • Implement remote job reconciliation for the Kratos job model. The SDK's V1 flow registers the job before uploading data and writes the _SUCCESS marker last, so a kill in that window strands the job in PENDING TEMP BUCKET UPLOAD with data in S3 that never ingests. This is the primary root cause in nvbug 6705092, where 64 stranded IDs were found in a two-day sample.

  • Note for the design: SDK V2 reverses the ordering, uploading all data and the _SUCCESS marker before registering the job, which makes the stranded-PENDING state structurally impossible. V2 is selected only for the china profiles and cannot be forced on global-prod through the SDK's public API.

Tests

  • Same backend conformance suite as the object-store backend.
  • Storage-only mode reaches terminal success at SUCCESS TEMP BUCKET UPLOAD without waiting for ingestion.
  • Ingestion mode polls through the intermediates to SUCCESS.
  • Indeterminate submit retains and quarantines rather than resubmitting.
  • Environment enum rejects production and staging.
  • A build without the Kratos tag omits the SDK and reports the backend as unavailable.

Open question

Whether global-prod can honor a client-supplied upload ID. V1 already transmits a client-generated UUID that the server discards, so this may be a server-side configuration question rather than an API change. Resolving it removes the ambiguous-outcome window entirely.

Parent: #1004

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing backend interface from #1047 and the generic object-store backend, then run the shared backend conformance suite. Trace GetUploadStatus and NewBulkUploadKratosOptionsWithHttpClient; done means the tagged and untagged builds behave as specified, all seven statuses and error classes are handled, configuration validation works, and the listed tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.