[Tracking] Fabric distributed LightGBM reliability: fixes, regressions, and runtime delivery

Nobody has claimed this yet.

## Summary

Track the work needed to make distributed LightGBM training reliable and diagnosable on Fabric, across the existing native-crash, topology, ingestion, and release-delivery issues.

A customer reports recurring executor/network failures, including `connection refused`, across multiple Fabric projects. A single-partition fit is substantially more reliable than distributed training. The customer has already tried caching, different partitioning strategies, task/thread/batch settings, voting, bagging, cluster sizes, disabling dynamic allocation, and the `train.limit()` workaround.

This is a coordination-only umbrella, not a claim that those symptoms have one root cause. The customer's exact Fabric runtime build, loaded SynapseML/native artifacts, first-failure logs, and minimal reproduction have not been supplied. Existing leaf issues remain the canonical places for reproductions and fixes. Similar final error messages are not grounds for closing them as duplicates.

The closest existing reports are microsoft/SynapseML#2310 and microsoft/SynapseML#2438. A separate Fabric report, [lightgbm-org/LightGBM#6974](https://github.com/lightgbm-org/LightGBM/issues/6974#issuecomment-3121202688), was routed to microsoft/SynapseML#2310, not resolved by a native fix. This umbrella adds Fabric runtime delivery and cross-family acceptance tracking; it does not replace those issues or the broader production-readiness discussion in microsoft/SynapseML#1956.

## Why more parameter tuning is not a sufficient answer

SynapseML coordinates Spark tasks, but LightGBM trains through a native distributed collective with a fixed participating worker set. A native crash, memory exhaustion, or lost executor can leave surviving or retried tasks reporting communication failures.

Current [NetworkManager.scala](https://github.com/microsoft/SynapseML/blob/8c7143875c843c649a817cf3e8ba9c7bee23689c/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/NetworkManager.scala) distinguishes first-attempt connection failures from failed partial retries. It also supports fresh topology rounds for coordinated barrier-stage retries. Neither mechanism makes an in-progress native collective elastic or fixes a deterministic native crash.

The [SynapseML overview](https://github.com/microsoft/SynapseML/blob/8c7143875c843c649a817cf3e8ba9c7bee23689c/docs/Explore%20Algorithms/LightGBM/Overview.md#dynamic-allocation-limitations) already documents executor-topology limitations. The remaining work is to connect that guidance to known defects, supported runtime configurations, and the artifacts customers actually receive.

## Related failure families

| Failure family | Existing reports | Current status and scope |
| --- | --- | --- |
| Connection refused / network initialization | microsoft/SynapseML#2310 | Open and canonical. Inspect the first failed attempt before diagnosing a firewall or increasing socket timeouts. A retry-time connection error alone does not identify the original fault. |
| Distributed categorical native crashes | microsoft/SynapseML#2697, with related reports microsoft/SynapseML#2302, microsoft/SynapseML#2438, microsoft/SynapseML#2467, microsoft/SynapseML#2674 | The tracker is open. The upstream split-buffer correction is a candidate requiring controlled native A/B confirmation, not a proven explanation of this customer's report. |
| Streaming-mode native ingestion corruption | microsoft/SynapseML#2333 | Open. Keep separate from categorical split-buffer crashes. Bulk mode or reduced ingestion concurrency can be diagnostic controls, not a proven general fix. |
| Fabric sampled-reference Dataset construction | microsoft/SynapseML#2242 | Reopened. The [latest reproduction](https://github.com/microsoft/SynapseML/issues/2242#issuecomment-5548303295) fails with explicitly unique feature names, before those names are applied. Supplied-name deduplication does not resolve that path. |
| Intermittent hangs and repeated-fit failures | microsoft/SynapseML#1967, microsoft/SynapseML#2323, microsoft/SynapseML#2287 | Open. Socket lifecycle improvements do not establish that every initiating cause has been fixed. Separate sequential-fit, concurrent-fit, ingestion, and native-training failures. |
| Executor replacement and dynamic allocation | microsoft/SynapseML#319, microsoft/SynapseML#1618 | Unresolved architectural limitation. Disabling autoscaling does not prevent OOM, native crashes, or other executor loss. |
| Socket resets around early stopping | microsoft/SynapseML#728 | Open. Retain as a distinct investigation unless first-failure evidence establishes a common cause. |

Historical reports microsoft/SynapseML#748, microsoft/SynapseML#949, and microsoft/SynapseML#2044 were closed as duplicates of microsoft/SynapseML#2310, not as fixed. microsoft/SynapseML#2467 was closed without a demonstrated SynapseML correction. Neither kind of closure proves the customer-facing failure is resolved.

The `limit().cache().count()` workaround appears in microsoft/SynapseML#2438. Compare equivalent input snapshots and physical plans before treating it as a repair; changes in rows, partition layout, and recomputation can all change whether a failure appears.

## Relevant fixes already merged

All PRs below merged into `master`. Dates are UTC. These are separate corrections, not a single fix for every networking symptom.

| PR | Merge date | What it fixes, and what it does not |
| --- | --- | --- |
| [microsoft/SynapseML#2593](https://github.com/microsoft/SynapseML/pull/2593) | 2026-08-06 | Preserves partition topology during row counting, fixing the AQE/count-array mismatch in microsoft/SynapseML#2278. |
| [microsoft/SynapseML#2595](https://github.com/microsoft/SynapseML/pull/2595) | 2026-08-07 | Retains worker-port reservations until native initialization, fixing the race reported in microsoft/SynapseML#2230. |
| [microsoft/SynapseML#2612](https://github.com/microsoft/SynapseML/pull/2612) | 2026-08-12 | Improves first-failure diagnostics, socket cleanup, and coordinated barrier topology retries. Does not establish that microsoft/SynapseML#2310 or native-crash reports are resolved. |
| [microsoft/SynapseML#2508](https://github.com/microsoft/SynapseML/pull/2508) | 2026-08-12 | Deduplicates supplied feature names. The distinct reference-construction failure in reopened microsoft/SynapseML#2242 remains. |
| [microsoft/SynapseML#2637](https://github.com/microsoft/SynapseML/pull/2637) | 2026-08-16 | Fixes IPv6 endpoint handling and transport for microsoft/SynapseML#2152. Relevant only when the endpoint/signature matches. |
| [microsoft/SynapseML#2662](https://github.com/microsoft/SynapseML/pull/2662) | 2026-08-20 | Corrects failed streaming Dataset ownership/cleanup. Explicitly does not resolve the corruption reported in microsoft/SynapseML#2333. |
| [microsoft/SynapseML#2664](https://github.com/microsoft/SynapseML/pull/2664) | 2026-08-27 | Replaces driver-collected validation broadcast arrays with bounded transfer for microsoft/SynapseML#2294. Driver disk and native-worker memory requirements remain. |
| [microsoft/SynapseML#2695](https://github.com/microsoft/SynapseML/pull/2695) | 2026-09-04 | Propagates iteration failures instead of presenting partial models as successful fits. Does not prevent a native crash that terminates an executor. |

## Merged does not mean delivered to Fabric

Evidence reviewed on 2026-09-05:

- The published [SynapseML v1.1.3 release](https://github.com/microsoft/SynapseML/releases/tag/v1.1.3) dates from April 2026. Its source predates the recent topology, port, validation-transfer, and iteration-error work.
- The public [Fabric Runtime 1.3 component manifest](https://github.com/microsoft/synapse-spark-runtime/blob/09c1d34b9b467a6bba9a0474046784c7860d3d26/Fabric/Runtime%201.3%20%28Spark%203.5%29/Components.json) lists `synapseml-lightgbm` as `2.12-1.1.3` and `lightgbmlib` as `3.3.510`. This is a published baseline, not proof of what the reporting customer's session loaded.
- The [Fabric Runtime 2.0 manifest](https://github.com/microsoft/synapse-spark-runtime/blob/47dba243f03075fec01b44d982ad532f3abf2123/Fabric/Runtime%202.0%20%28Spark%204.1%29/Components.json) lists the Python package as `1.1.3+spark4.1`. Python package metadata alone does not establish which native binaries are loaded. Track Spark/Scala-compatible artifacts separately rather than carrying the master artifact into every runtime.
- Both [v1.1.3](https://github.com/microsoft/SynapseML/blob/a833941704b5e8334ddb40a9d601d7e0c7c0ce9f/build.sbt) and [current master](https://github.com/microsoft/SynapseML/blob/8c7143875c843c649a817cf3e8ba9c7bee23689c/build.sbt) still reference `com.microsoft.ml.lightgbm:lightgbmlib:3.3.510`.
- The distributed split-buffer correction in [lightgbm-org/LightGBM#6738](https://github.com/lightgbm-org/LightGBM/pull/6738) merged upstream and appears in [LightGBM v4.6.0](https://github.com/lightgbm-org/LightGBM/releases/tag/v4.6.0). That does not establish its delivery in SynapseML's separate native/SWIG artifact. Installing a newer Python `lightgbm` package does not replace the native libraries loaded through SynapseML's JVM integration.

## Work to track

- [ ] Resolve the native-crash families in their existing leaf issues. For microsoft/SynapseML#2697, complete controlled native A/B confirmation, then publish and consume a matched native/SWIG correction or compatible native upgrade. Record source/build provenance; do not substitute only one of the paired native libraries.
- [ ] Publish a fix-to-release-to-Fabric-runtime matrix for the merged SynapseML corrections. Distinguish current master, a released Spark/Scala-compatible artifact, the Fabric runtime/channel that includes it, and the versions actually loaded on driver and executors.
- [ ] Provide a supported distributed-training baseline and a troubleshooting path in SynapseML documentation, with corresponding Fabric documentation follow-through. Cover task slots versus native workers, fixed executor membership, native memory headroom, barrier compatibility, materialized input, and concurrent-fit limits. Do not describe `repartition(1)` as a distributed solution.
- [ ] Make the first failure discoverable through fit/stage/task-attempt identifiers, actual worker topology, native-library provenance, and executor exit diagnostics. Confirm that failures propagate rather than returning partial models, and that cleanup permits subsequent fits.
- [ ] Add repeatable multi-executor coverage on supported Fabric runtime builds with identified artifacts. Include two and four executors, repeated sequential fits, streaming and bulk, dense and sparse inputs, categorical regressions from the leaf issues, validation/early stopping, and model save/load. Test concurrent fits separately if that usage is supported.
- [ ] Exercise controlled executor loss and stage restart separately from successful training. Document supported whole-stage recovery and unsupported partial-task recovery. Require bounded failure with the original cause where recovery is not supported.

## Safe interim guidance

Use one fit at a time while diagnosing, a stable executor set, and an explicit task count that fits available concurrent task slots. `numTasks` counts Spark training tasks, not a guaranteed number of distinct executor hosts. Confirm the actual native worker topology in logs.

Materialize a consistent input snapshot and inspect partition sizes and memory headroom before increasing parallelism. Repartitioning by an arbitrary categorical key can introduce skew. Ranker query-group requirements are a separate consideration.

Compare streaming and bulk only with enough executor/native memory for bulk's additional copies. `numThreads=1` controls native training threads and is not proof that Spark ingestion has become single-threaded. Voting, bagging, categorical-threshold changes, and longer timeouts do not repair native memory corruption. `numBatches` changes how training proceeds and should not be presented as an equivalent one-pass fit.

For the customer-specific diagnosis, collect the runtime build/channel, Spark and Scala versions, loaded SynapseML and `lightgbmlib` provenance, model parameters, executor/task/core counts, input and partition sizes, categorical cardinalities, physical plan, and first-failure executor logs before Spark retries. Include native crash logs when available, after removing credentials, customer data, and sensitive identifiers.

## Completion criteria

A documented multi-executor configuration must work repeatedly on identified Fabric builds without reducing all training to one partition. Each linked failure family must have a delivered correction, a verified non-applicability decision, or an explicitly documented limitation. A green run on one small dataset, a merged PR alone, or a suppressed exception is not sufficient.

This is a coordination umbrella rather than a single implementation task. Start with the canonical leaf issues, especially #2310 and #2697, then read lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/NetworkManager.scala, the LightGBM overview, and the Fabric Components.json manifests. Done requires the tracked fixes, runtime-delivery matrix, documentation, diagnostics, and multi-executor regression coverage to be completed across their respective issues.

Written by the indexing model from the issue text.