mudler / mudler/LocalAI

FlashAttention2 not available in hipblas docker images

Open
#9,138 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/backends bug unconfirmed upstream issue waiting-from-reporter
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

LocalAI version:

LocalAI v4.0.0 (8e8b7df715e620b64d07cddf6b73bff8f966dac5)

Environment, CPU architecture, OS, and Version:

AMD Strix Halo 395+
128Gb Unified Memory (96 VRAM)
Fedora 42

Describe the bug

I am unable to load either qwen3-tts or qwen3-asr with errors saying FlashAttention2 is not found.

To Reproduce

Docker Compose file:

  localai:
    container_name: localai
    image: localai/localai:latest-gpu-hipblas
    environment:
      DEBUG=true
      MODELS_PATH: /models
    ports:
      - "8080:8080"
    volumes:
      - ./backends:/backends
      - ./config:/config
      - ./models:/models
    devices:
      - /dev/dri:/dev/dri
      - /dev/kfd:/dev/kfd
    group_add:
      - "video" 
    restart: unless-stopped

After that install qwen3-tts and try to run the model.
Backend will download but fail load.

Expected behavior

Models should load without errors.

Logs

Image

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the localai/localai:latest-gpu-hipblas Docker Compose configuration and the qwen3-tts or qwen3-asr model. Inspect how the hipblas image is built and how its backend dependencies are installed. Done means the affected models load successfully in that image without the FlashAttention2 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
build-system, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.