[BUG] Distributed categorical LightGBM crashes in SerialTreeLearner::SplitInner

### SynapseML version

Current `master` and released versions that package
`com.microsoft.ml.lightgbm:lightgbmlib:3.3.510`.

### System information

- **Language versions:** Python/Scala; reports span Java 8, 11, and 17
- **Spark versions:** Multiple Spark generations
- **Spark platforms:** Microsoft Fabric, Azure Synapse, and Databricks
- **Required topology:** The failure is reported with two or more LightGBM workers; equivalent single-worker runs succeed

### Describe the problem

This is an umbrella issue for a probable native distributed categorical-training
defect currently split across #2302, #2438, #2467, and #2674.

The common primary failure is a native process crash in:

```text
LightGBM::SerialTreeLearner::SplitInner(...)+0xf7b
```

After one executor exits, surviving workers can report `Connection refused`,
`Socket recv error, code: 104`, or connection resets. Those messages are
secondary symptoms of the lost native worker, not necessarily the initiating
network failure.

The strongest reproduction is #2674:

- one executor succeeds;
- two or more executors succeed with categorical cardinality 56;
- the same distributed workload fails above that boundary;
- changing only `lightgbmlib` from `3.3.510` to `3.3.500` reportedly restores
success.

This aligns closely with lightgbm-org/LightGBM#6491. Upstream found that
distributed categorical split metadata used an undersized communication buffer
after two `int64_t` fields were added to `SplitInfo`. That can overwrite native
memory and crash in `SerialTreeLearner::SplitInner`. The buffer-size correction
was merged in lightgbm-org/LightGBM#6738 and released in LightGBM 4.6.0.

SynapseML still packages `lightgbmlib:3.3.510`. The available binary evidence
shows that `.510` contains the quantized-gradient feature associated with the
upstream `SplitInfo` change, but its manifest does not identify the exact source
commit. A controlled native A/B test is therefore required before declaring
the upstream defect to be the confirmed root cause.

#### Related SynapseML reports

| Issue | Evidence |
| --- | --- |
| #2302 | Multi-executor-only SIGSEGV in `SerialTreeLearner::SplitInner+0xf7b` |
| #2438 | Fabric executor exit 134 with the same native frame; retries may eventually succeed |
| #2467 | Java 17 multi-executor crash with the same frame; reporter says LightGBM 4.7 succeeds on the same data |
| #2674 | Synthetic cardinality boundary and reported `3.3.500` versus `3.3.510` A/B |

#### Relevant fixes already on `master`

These improve adjacent failure handling but do not correct this native crash:

- #2612 preserves the first task failure and explains secondary retry-time
connection refusals.
- #2662 fixes native Dataset cleanup leaks but explicitly does not resolve
native training corruption.
- #2695 propagates native iteration exceptions instead of returning a partial
model as successful training. A SIGSEGV terminates the executor before Scala
can catch it.
- #2596 can reduce category cardinality through top-K lumping, but that changes
feature semantics and is only a possible workaround.

There is no merged SynapseML change that backports upstream LightGBM #6738 or
upgrades the matched native/SWIG integration to LightGBM 4.6 or newer.

### Code to reproduce issue

Use the synthetic notebooks attached to #2674 as the initial canonical
reproduction. They hold the Spark topology and model parameters constant while
changing categorical cardinality:

1. Run on a fixed cluster with at least two executors and with autoscaling,
dynamic allocation, speculation, and executor decommissioning disabled.
2. Fit the supplied distributed `LightGBMRegressor` workload with 56 categories.
3. Repeat with 57 or more categories.
4. Capture the first executor's `hs_err_pid*.log`, rather than only the final
peer socket error.

The reproduction should then be converted into a repository-owned,
multi-executor regression test that does not depend on the attached notebooks.

### Proposed investigation and acceptance criteria

- [ ] Reproduce the 56/57-category boundary with the exact
`lightgbmlib:3.3.510` jar and record its SHA-256.
- [ ] Run the same test with `3.3.500` as a diagnostic baseline.
- [ ] Build a matched native and SWIG pair containing upstream LightGBM #6738,
or a supported LightGBM 4.6+ integration, and rerun the test.
- [ ] Cover both classifier and regressor categorical training where practical.
- [ ] Cover bulk and streaming transfer modes to distinguish this family from
the separate streaming multi-core corruption tracked by #2333.
- [ ] Verify that reusable reference Dataset behavior introduced with `.510`
still works; do not ship an unqualified downgrade to `.500`.
- [ ] Add a permanent multi-worker regression test and document the first-error
diagnostic pattern.
- [ ] Link and triage #2302, #2438, #2467, and #2674 against the confirmed
result.

### Other info / logs

The identical native symbol and offset across multiple Java versions makes a
JVM defect unlikely. The distributed-only categorical trigger, the reported
`.500`/`.510` A/B result, and the matching upstream fix make this a genuine,
high-priority candidate for the customer reports where single-node execution is
stable but distributed training eventually loses an executor.

### What component(s) does this bug affect?

- [x] `area/lightgbm`: Lightgbm project

### What language(s) does this bug affect?

- [x] `language/scala`: Scala source code
- [x] `language/python`: Pyspark APIs

### What integration(s) does this bug affect?

- [x] `integrations/synapse`: Azure Synapse integrations
- [x] `integrations/databricks`: Databricks integrations

Start with the synthetic notebooks attached to #2674 and reproduce the 56/57-category boundary using the exact lightgbmlib:3.3.510 jar, recording the first executor's hs_err_pid*.log. Compare 3.3.500 with a matched native/SWIG build containing upstream LightGBM #6738 or LightGBM 4.6+, then cover classifier and regressor modes where practical. Done means a repository-owned multi-executor regression test and documented first-error diagnostics, while preserving .510 Dataset behavior.

Written by the indexing model from the issue text.