Feature Request: Support for Bonsai-8B 1-bit Quantized Model

Feature Request: Support for Bonsai-8B 1-bit Quantized Model

Hello,

I'd like to request support for Prism-ML's Bonsai-8B, a groundbreaking 1-bit quantized model available on Hugging Face:

Model URL: https://huggingface.co/prism-ml/Bonsai-8B-gguf

Why This Matters

Bonsai-8B represents a significant advancement in efficient LLM deployment:

• Compact Size: An 8B parameter model compressed to just ~1.15 GB
• Impressive Performance: Achieves an MMLU-R of 65.7, competitive with larger models
• Energy Efficient: Reported to be 5x more energy efficient than comparable 8B models
• Dramatically Smaller: 14x smaller footprint than typical 8B models

NPU Potential

The 1-bit quantization approach makes Bonsai-8B an ideal candidate for NPU acceleration. The dramatically reduced memory footprint and simplified computations could enable:

• Smooth inference on edge devices with NPUs
• Significantly reduced power consumption
• Real-time AI capabilities without cloud dependency
• Privacy-preserving local inference

Request

I'd love to see FastFlowLM add support for this model. Given its efficiency characteristics, it seems like a perfect fit for demonstrating NPU performance advantages with quantized models.

Thank you for considering this request!

Best regards

Start by reviewing how FastFlowLM currently supports quantized models and then inspect the Bonsai-8B-gguf model at the linked Hugging Face URL. Done should mean FastFlowLM supports loading and running this model, including the requested NPU inference path, but the issue does not identify files, tests, or an implementation entry point.