beehive-lab / beehive-lab/GPULlama3.java

Support the Qwen3.5 architecture

Open
#143 0 comments 0 reactions 1 assignee Claimed by @orionpapadakis View on GitHub
Dominant language
Java
Stars
274
Forks
37
Avg merge
5d 1h
Merged PRs (30d)
2

Description

**Is your feature request related to a problem? Please describe.**
When I run the model `Qwen3.5-4B-Q4_K_M.gguf`, I hit a NullPointerException at this line:
```
int modelContextLength = (int) metadata.get("qwen3.context_length");
```
in `Qwen3Configuration.createConfiguration`.

**Describe the solution you'd like**
Apparently, the structure of the Qwen3.5 is different to Qwen3. It would be nice if the code recognised this.

**Describe alternatives you've considered**
`llama.cpp` handles `Qwen3.5-4B-Q4_K_M.gguf` fine:
```
docker run --rm -it --gpus all -p 8080:8080 -v /home/henryp/Downloads/:/models ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen3.5-4B-Q4_K_M.gguf --jinja --host 0.0.0.0 --port 8080
```

**Additional context**
I ran:
```
$ ./llama-tornado --gpu --verbose-init --ptx --model /home/henryp/Downloads/Qwen3.5-4B-Q4_K_M.gguf --heap-max 20g --prompt "Explain the benefits of GPU acceleration."
Detected TornadoVM backend: ptx (from /usr/local/bin/Java/Tornado/etc/tornado.backend)
WARNING: Using incubator modules: jdk.incubator.vector
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "java.util.Map.get(Object)" is null
at org.beehive.gpullama3.model.loader.Qwen3ModelLoader.createConfiguration(Qwen3ModelLoader.java:49)
at org.beehive.gpullama3.model.loader.Qwen3ModelLoader.createConfiguration(Qwen3ModelLoader.java:29)
at org.beehive.gpullama3.model.loader.AbstractModelLoader.loadModel(AbstractModelLoader.java:94)
at org.beehive.gpullama3.model.ModelType$5.loadModel(ModelType.java:58)
at org.beehive.gpullama3.model.loader.ModelLoader.loadModel(ModelLoader.java:99)
at org.beehive.gpullama3.LlamaApp.main(LlamaApp.java:61)
Error: Command failed with return code 1
$ java --version
java 21.0.1 2023-10-17
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19, mixed mode, sharing)
14:45:31 henryp@thinkpad ~/Code/Java/GPU/GPULlama3.java
$ tornado --version
version=4.0.2-jdk21-dev
branch=master
commit=eab0b59

Backends installed:
- ptx
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.