microsoft / microsoft/SynapseML

feat: ship a GPU-enabled LightGBM native so deviceType=gpu can actually train

Open
#2,642 2 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

#2627 adds a `deviceType` parameter so `device_type=gpu|cuda` can be set on the LightGBM learners. The parameter works — it reaches the native layer, verified by LightGBM rejecting it *by name*. But it cannot actually train on a GPU, because the native library SynapseML ships is CPU-only.

That is the other half of GPU support, and it is a build problem rather than a code problem.

## Evidence

`build.sbt` pins `com.microsoft.ml.lightgbm:lightgbmlib` `3.3.510`. Scanning the bundled `lib_lightgbm.so`:

```
OpenCL 0 occurrences
clCreateKernel 0 occurrences
cuda_tree_learner 0 occurrences
```

Training with `device_type=gpu` fails with `GPU Tree Learner was not enabled in this build.`, and `cuda` with the CUDA equivalent. Those strings are compiled into the shipped binary.

## No dependency bump fixes this

- `com.microsoft.ml.lightgbm:lightgbmlib` publishes one artifact and `3.3.510` is the newest that exists (19 versions, newest 2023-05-15). We are already on it.
- Every other `lightgbm` artifact on Maven Central is either older, a PMML/conversion tool, or a different binding.
- The one newer binding, `io.github.metarank:lightgbm4j` (LightGBM 4.4.0), ships CPU-only natives too. Its README is explicit that GPU requires you to rebuild LightGBM yourself with `-DUSE_CUDA=1 -DUSE_SWIG=ON` and point `LIGHTGBM_NATIVE_LIB_PATH` at the result.

So there is no artifact to switch to. Someone has to build the native.

## What this would involve

1. Build LightGBM from source with `-DUSE_GPU=1` (OpenCL) and/or `-DUSE_CUDA=1`, plus `-DUSE_SWIG=ON`, for each platform we support.
2. Decide how it ships. A GPU build needs an OpenCL/CUDA runtime present at load time, so it likely cannot simply replace the current native in the main jar without breaking CPU-only users — a classifier, a separate artifact, or runtime selection with fallback.
3. Add a load path that prefers the GPU native when present and falls back to the CPU one, so nothing regresses for users without a GPU.
4. Get GPU CI capacity, or accept that this stays manually verified.

## Why it is worth doing separately

Step 2 is the real design question and it affects packaging for every SynapseML user, GPU or not. It should not ride along inside a parameter PR.

Worth noting the ordering is fine either way: #2627's GPU tests assert that a device request is *never silently downgraded to CPU*, accepting either a successful fit or a failure naming the tree learner. On a GPU-capable native they start passing by training instead of by failing, with no test changes needed.

Contributor guide

Open the contributing guide

Research direction

Start with build.sbt and inspect how the bundled lib_lightgbm.so is packaged and loaded. Review the GPU tests from #2627 and determine the artifact or runtime-selection design, including CPU fallback. Done means supported GPU natives can train without silently downgrading, while CPU-only users continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, scala
Domain
build-system, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.