Future of `candle-transformers` / long-term plans

Hi there!

Apologies for the vague issue title, but I was struggling to think of one that conveyed my sentiments.

I'm the primary maintainer of [rustformers/llm](https://github.com/rustformers/llm), which implements several common LLM architectures atop GGML as an easy-to-use library. In some sense, it can be considered a robust, consistent and extensible Rust library-ification of `llama.cpp` with support for other architectures.

Recently, I've been considering [winding down development on it](https://github.com/rustformers/llm/discussions/435) in favour of encouraging people to use `candle-transformers` instead, because Candle can evolve faster than we do, supports more models, and isn't held back by the free time I/our contributors have.

My initial plan was to get `llm` back up to speed with the latest in `llama.cpp` and elsewhere, and then add support for other backends, so that Candle could be a secondary backend and (likely) become the primary backend in future. However, chasing a moving target is quite difficult, and `candle-transformers` already covers much of the same ground.

With that in mind, people in that discussion have raised a few issues around switching to `candle-transformers` and think `llm` is still relevant. I think it'd be simpler for the ecosystem if there was a single place for LLMs, but the concerns raised have made it harder to provide a straightforward recommendation to switch to `candle-transformers`.

So, here are my questions:

1) What's the long-term plan for `candle-transformers`? Vague question, I know, but will it live in this repo forever? Will it become an ecosystem unto itself like Python `transformers`?
2) Will the models share a unified interface to make it easy to swap between architectures?
3) Similarly, are there any plans to offer a higher-level abstraction for model inference? I'm not *completely* happy with our own interface, but it makes common tasks (prompt-feeding, streamed inference) easy, and harder tasks (custom inference, custom models) doable:
4) Are there plans for Metal acceleration support in the near-future? I'm guessing this is tracked in #313, but I'm not sure what the timeline of that is.
5) Finally, do you think a library like `llm` is necessary? My gut feeling is that Candle (and `candle-transformers`) could grow to cover all of its territory, but there may still be some value in custom high-level abstractions and support for other backends. That would be obviated by Candle developing its own high-level abstractions and increasing its backend support, though.

Sorry about the wall of questions, but your input is hugely useful in figuring out our own direction. Knowing what you have planned will clarify some of the unknowns for us and let us figure out what to do next.

Thanks in advance! ❤️

Start with the linked rustformers/llm discussion and crates/llm/examples/inference.rs, then review issue #313 for the Metal acceleration question. The issue names no implementation file or test; done would be a maintainer-backed roadmap addressing the listed architecture, interface, backend, and ecosystem questions.

Written by the indexing model from the issue text.