Synchronize inferred target count across distributed workers
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
### Description
`ObjFunction::Targets(const MetaInfo&)` is evaluated independently on each distributed worker. For objectives that infer the number of regression targets from `info.labels.Shape(1)`, this can return different values on different workers.
An empty distributed worker can receive a label-less DMatrix with label shape `(0, 0)`. Multi-target objectives generally use a single-target fallback for this shape, while a non-empty worker sees labels shaped `(n_rows, K)`:
- non-empty worker: `Targets(info) == K`
- empty worker: `Targets(info) == 1`
The learner currently uses this local result to size `num_target`, the model outputs, and the intercept. Workers can therefore initialize incompatible model state before a later collective exposes the mismatch.
### Example
Train a multi-target regression objective on two workers without explicitly setting `num_target`:
- worker 0 has rows with three-column labels;
- worker 1 has no local partition and constructs an empty, label-less DMatrix.
The workers infer three and one targets respectively. Depending on the subsequent code path, this can produce incompatible intercept sizes, mismatched collective buffers, a collective failure, or a hang.
This affects objectives whose target count is inferred from the local label shape. Single-output objectives and objectives whose output count comes from shared configuration are unaffected.
### Expected behavior
All workers should agree on the target count before model and intercept initialization. Empty workers should act as having an unknown target count rather than asserting one target, while conflicting target counts from non-empty workers should be rejected collectively.
Possible longer-term alternatives include preserving `(0, K)` label shapes for empty distributed partitions or passing the learner's authoritative output count explicitly to objective methods.
This was exposed while implementing exact distributed intercepts in #12557, but the underlying target-discovery problem is learner-level and independent of radix selection.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at ObjFunction::Targets(const MetaInfo&) and trace the learner path that sizes num_target, model outputs, and the intercept, using the distributed intercept work in #12557 for context. Done means workers agree on the inferred target count before initialization, empty workers remain unknown, and conflicting non-empty counts are rejected collectively.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100