Investigate integrating a local language model
- Dominant language
- Swift
- Stars
- 17
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Highly quantized language models that can run locally are getting more and more popular with even [Chrome shipping a Gemini Nano model](https://developer.chrome.com/docs/ai/built-in) in their latest canary builds. Models like [Phi-3-mini](https://github.com/microsoft/Phi-3CookBook) already achieve impressive performance for being comparatively small and [support cross-platform inference](https://github.com/microsoft/Phi-3CookBook/blob/fb35e596083b05f35ddc73bf0d6936effb67f16f/md/03.Inference/Rust_Inference.md) using [a Rust library named `candle`](https://github.com/huggingface/candle).
It would be cool if we could bundle such a model with D2, e.g. as a command and/or as a [`Conversator`](https://github.com/fwcd/d2/blob/71acc52c7ede483eeee6b2571bfb5a88d3b653c2/Sources/D2Commands/Misc/Conversator.swift).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading Sources/D2Commands/Misc/Conversator.swift and the linked candle Rust inference example. Investigate how a highly quantized local model such as Phi-3-mini could be bundled with D2 and whether the integration belongs in a command, a Conversator, or both. Done requires a defined integration approach and a working local-model feature.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, swift
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100