beehive-lab / beehive-lab/GPULlama3.java

[POC] Minimal Kernel Autotuning Support

Open
#96 0 comments 0 reactions 1 assignee Claimed by @mikepapadim View on GitHub
enhancement
Dominant language
Java
Stars
274
Forks
37
Avg merge
5d 1h
Merged PRs (30d)
2

Description

### Summary

Add **minimal kernel autotuning support** to TornadoVM to automatically evaluate and select efficient execution configurations (e.g., work-group and grid sizes) at runtime, inspired by Triton’s `triton.autotune`.

---

### Scope (Intentionally Small)

This sub-issue focuses on a **first, minimal autotuning capability**:

- Support autotuning for **work-group and grid dimensions only**
- Limit autotuning to **single-kernel task graphs**
- Perform autotuning **once per kernel per device**
- Cache the best configuration in-memory (no persistence required)

This is intended as a foundation for future extensions (tiling, memory layouts, heuristics).

---

### Proposed Functionality

1. **Configuration Set**
- Allow a small, user-defined set of candidate execution configurations
- Example: different `(globalSize, localSize)` combinations

2. **Runtime Benchmarking**
- On first execution, run each configuration and measure execution time

3. **Selection & Caching**
- Select the fastest configuration
- Cache the result for subsequent executions on the same device

4. **Transparent Integration**
- Autotuned configuration replaces the default execution without user-side changes

---

### Motivation

Performance in TornadoVM is sensitive to execution parameters and GPU architecture.
Today, finding good configurations is manual and hardware-specific.

Even a limited autotuning mechanism would:
- Reduce manual tuning effort
- Improve out-of-the-box performance
- Benefit GPU-heavy workloads (e.g., attention, GEMM) used in **GPULlama3**

---

### Example (Hypothetical)

A kernel is executed with candidate local sizes:
- `(16,16)`
- `(32,8)`
- `(8,32)`

TornadoVM benchmarks each once, selects the fastest, and reuses it for all future executions.

---

### Out of Scope (For This Sub-Issue)

- Persistent caching across JVM runs
- Large configuration search spaces
- Compiler-driven autotune generation
- Memory tiling or algorithmic variants

---

### Expected Outcome

A small but functional autotuning mechanism that demonstrates feasibility and provides immediate performance benefits, serving as a stepping stone toward a full Triton-like autotuning framework.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.