feat(request-trace-uploader): add Kratos Bulk Upload backend
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Why
Internal customers export request records to Kratos; external customers cannot reach Kratos at all and need the generic object store. The uploader must serve both from one image by changing only the backend setting.
Kratos Bulk Upload is not an S3 endpoint. It is a job API that internally vends short-lived STS credentials and PUTs to S3, then reports terminal status asynchronously. It is reached through the internal Go SDK at golang.nvidia.com/kratos/kratos-go-sdk, which is NVIDIA-internal and unlicensed.
Scope
-
Implement the Kratos backend against the existing backend interface from #1047. No new interface, no second uploader implementation.
-
Place the implementation behind a build tag so the core and the generic object-store backend compile without the internal SDK. A build without the tag must report that the Kratos backend is not compiled in, rather than failing to build.
-
Submit with
UploadContext(..., kratosWaitForCompletion: false)and poll the exportedGetUploadStatusseparately, so submission and confirmation stay independent steps. -
Own the terminal-status map rather than using the SDK's
checkUploadStatus. That helper only breaks onSUCCESSorFAILED BULK UPLOAD INGESTION, so in storage-only mode it polls to its 1800s ceiling and reports a timeout on a job that actually succeeded atSUCCESS TEMP BUCKET UPLOAD. -
Model all seven Kratos statuses, including the
PENDING TEMP BUCKET UPLOAD,PENDING BULK UPLOAD INGESTION, andPROCESSINGintermediates that the shell uploader ignores. -
Declare capabilities honestly. Kratos V1 mints the upload ID server-side, so resubmit safety is false and indeterminate submissions must resolve to retain-and-quarantine rather than resubmit. V2 accepts a client-supplied upload ID; the SDK currently selects V2 only for the china profiles.
-
Classify Kratos errors into the retryable, terminal, and ambiguous classes and into segment or backend scope.
-
Supply the uploader's own HTTP client through
NewBulkUploadKratosOptionsWithHttpClientso Retry-After handling and per-call latency instrumentation are possible. -
Bound the SDK's internal 100-attempt S3 retry configuration with the configured per-attempt timeout via context.
-
Configuration: namespace, environment, ingestion toggle, database and table, mode, ACE ID, S3 endpoint, region, status poll interval and deadline. Environment accepts only
global-prod,china-production, andchina-staging; reject the shell uploader'sproductionandstagingvocabulary explicitly at startup. -
Credentials come from the read-only mount by key name, never from values in configuration.
-
Implement remote job reconciliation for the Kratos job model. The SDK's V1 flow registers the job before uploading data and writes the
_SUCCESSmarker last, so a kill in that window strands the job inPENDING TEMP BUCKET UPLOADwith data in S3 that never ingests. This is the primary root cause in nvbug 6705092, where 64 stranded IDs were found in a two-day sample. -
Note for the design: SDK V2 reverses the ordering, uploading all data and the
_SUCCESSmarker before registering the job, which makes the stranded-PENDING state structurally impossible. V2 is selected only for the china profiles and cannot be forced on global-prod through the SDK's public API.
Tests
- Same backend conformance suite as the object-store backend.
- Storage-only mode reaches terminal success at
SUCCESS TEMP BUCKET UPLOADwithout waiting for ingestion. - Ingestion mode polls through the intermediates to
SUCCESS. - Indeterminate submit retains and quarantines rather than resubmitting.
- Environment enum rejects
productionandstaging. - A build without the Kratos tag omits the SDK and reports the backend as unavailable.
Open question
Whether global-prod can honor a client-supplied upload ID. V1 already transmits a client-generated UUID that the server discards, so this may be a server-side configuration question rather than an API change. Resolving it removes the ambiguous-outcome window entirely.
Parent: #1004
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing backend interface from #1047 and the generic object-store backend, then run the shared backend conformance suite. Trace GetUploadStatus and NewBulkUploadKratosOptionsWithHttpClient; done means the tagged and untagged builds behave as specified, all seven statuses and error classes are handled, configuration validation works, and the listed tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100