🙋 Feature Wish: Optional per-token logprobs + a way to continue a session without full reprocessing (useful for speculative-decoding experiments)

🧩 Context

Hi FLM team! 👋 I'm a graduate researcher experimenting with heterogeneous
LLM inference on AMD Strix Halo (Ryzen AI MAX+ 395) — the idea is to run a
small "draft" model on the XDNA2 NPU and a larger "target" model on the iGPU,
following the speculative decoding (SD) pattern. FastFlowLM looks like the
only out-of-the-box way to get a model running on the NPU on Linux today 🙌,
so before going off to build something fully custom myself, I wanted to
check in here first and see if this is something you've already thought
about.

💭 What I'm hoping for

I haven't dug into the FLM source code, so I honestly don't know how big
these asks are, or whether they'd need to touch the NPU kernel side at all
— just raising them as a wishlist in case they're already on the roadmap,
or in case you can tell me whether they're realistic. 🙏

1. 📊 Per-token probabilities. Speculative decoding needs the draft
   model's probability for each proposed token to decide whether to accept or
   reject it. Right now /v1/chat/completions doesn't seem to return anything
   like logprobs / top_logprobs. Would it be possible to optionally expose
   this, similar to the OpenAI API convention?

2. ♻️ A cheaper way to continue an existing conversation. I noticed the
   docs mention CLI mode keeps a real KV cache across turns, while server mode
   reprocesses the full history on every request. For SD-style use, it would
   help a lot to have a way to say "continue from here, generate N more tokens"
   without resending/reprocessing everything — maybe via some kind of session
   id? I'm not sure how feasible this is given how the server is built
   internally, so I'd love to hear your thoughts.

🌱 Why this might be useful beyond my use case

Both of these would probably also help anyone trying to use FLM for
log-likelihood/perplexity evaluation or other research pipelines, not just
speculative decoding specifically.

Totally understand if this isn't a priority right now — just wanted to put
it out there! Thanks so much for FLM 💚, it's honestly been the easiest way
I've found to get anything running on the NPU on Linux.

Very Thanks Again! 🥰 HengJay

Start with the documentation for CLI mode, server mode, and the /v1/chat/completions entry point, then trace how conversation history and KV cache handling differ. Define acceptance criteria for optional per-token logprobs and session-based continuation, including API behavior and coverage for both capabilities; the issue does not name implementation files or tests.