NVIDIA / NVIDIA/nvcf

Document ModelExpress scale-out, verification, and fabrics

Open
#1,351 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 9h
Merged PRs (30d)
417

Description

Description

Complete the guide with the sections that require measured evidence: scale-out
behavior, verification, fallback, troubleshooting, and provider fabrics.

Definition of Done

  • Every support claim traces to an acceptance run.
  • Scale-out guidance reports observed behavior and the conditions it was measured
    under.
  • Verification tells readers how to prove the transport, not just that the
    workload is healthy.
  • Fallback behavior and its triggers are described, including that it is silent
    by design.
  • Troubleshooting covers the failures actually hit during acceptance.
  • EFA is documented as validated. InfiniBand and RoCE are documented from
    upstream behavior and labelled untested.

Resources

Three things the measurements changed about what the guide says:

  • Peer-to-peer is not uniformly faster. There is a crossover with worker count,
    so the guide describes the conditions and tells readers to measure at the
    fan-out they actually run, rather than asserting a benefit.
  • The usual precondition, that the hub is slower than the path between nodes, did
    not hold on this cluster. A CDN-backed hub inside the same cloud region was
    faster than a peer transfer, and it degraded far less under fan-out. The guide
    states this rather than repeating the assumption.
  • Fan-out is the variable that matters, not total worker count. One source serves
    a fixed aggregate bandwidth, so what matters is how many workers read from a
    source at once. The practical advice is to seed more than one source before a
    wide scale-out.

The guide carries no absolute numbers. They depend on model size, hub, fabric,
and scale, and quoting them without those conditions invites readers to plan
against figures that will not reproduce. Conditions and a measurement method
travel better.

Networking scope for this epic is EFA only. Testing every transport is not
required, and the guide says which fabric was tested.

Part of #1113.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ModelExpress guide and the acceptance runs behind its measurements. Document scale-out conditions, transport verification, fallback triggers, troubleshooting findings, and the tested EFA fabric; label InfiniBand and RoCE as upstream-based and untested. Done means every support claim traces to an acceptance run and the guide avoids unconditioned absolute numbers.

Written by the indexing model from the issue text.

Assessment

Domain
documentation, infrastructure, networking
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.