Deploy model on local hardware for inferencing

Here is a bit of an exploration - would it be feasible to deploy custom tuned, pruned, distilled models to people's phones - and the corresponding research - would it have good quality?

I have been thinking about deploying suggestions to computers and smartphones with either fairly reliable moderate latency internet or very intermittent internet. Here is my current best intermediate (2025) plan:

• for Platform.Bible, we can assume that they have at least moderate powered laptops, but very unreliable internet. The ideal may be hosting a pruned NLLB on device using sorbet and using LoRA for incremental tuning.
• for scriptureforge, because of the assumed low powered devices, the precomputed nmt suggestions may be best
• in either case, if they have a good enough connection to Dallas, live, full resolution nmt suggestions may be feasible, though the latency issues may still cause issues.

In the coming 6 months, we can do investigations into:

• ideal latency for Asia - and how it would impact user experience of nmt live suggestions
• how much of a KPMC hit is taken between live incremental suggestions full strength, precomputed suggestions and pruned live incremental suggestions, and even potentially pruned inference only, for phones. We don’t know right now what which one it will give us the smallest quality hit.
• checking the quality of incremental fine tuning with LoRA and seeing if it can be effectively run on CPU, real time on different hardware.
• seeing if inferencing of models can be run on a phone with typescript
• if need be, porting LoRA to different platforms such as dotnet (easy?) or typescript (harder?)

The issue names no files, tests, or entry points. Start by scoping the listed investigations across Platform.Bible and ScriptureForge, comparing live, precomputed, pruned, and on-device inference on the proposed hardware; done would require documented latency, quality, and LoRA feasibility results.