Lazy load models + Reload on demand

This is a potential feature request/question or maybe an opportunity for a new lib that builds on ML.NET. Hope you help me figure out which one of these.

We have a service used by N clients. Each of these clients uses (and frequently retrain) a single Engine (model). Clients also come and go so I can have N clients at one point and M a little while later where M != N. Sidenote, the number of clients could be anywhere between few tens to few hundreds or maybe thousands.

It is my understanding that fetching a prediction model from URI (that's our case, some people fetch from file system) and subsequent deserialization and setup is relatively expensive.

As far as I know, the prediction engine is also not thread-safe.

I assume that's why the PredictionEnginePool was created.

Our problem is the prediction engine pool is the only population during the app start. Also, there is some model auto-update functionality (watch) which doesn't work very well in our case as it introduces a lot of overhead. Imagine I have 1000 clients and I refetch 1000 models every X seconds.

What I am thinking about is having the ability to:
1. Add and remove model definitions from the pool on the fly.
2. Reload models (by name) from code (only when the model is retrained). Sidenote: This might need to be synced by some mechanism if there is more than one server (Redis, messaging, or something like that).
3. Have some sort of mechanism where only the most frequently used models are kept in memory.
a) Maybe timeout? E.g. when the model is requested and it doesn't exist, create it. Then keep it for X minutes with a moving window that is refreshed always when the same engine is requested again.
b) Probably introduce some fixed limit of concurrent models where if we run out of this limit, we simply drop the oldest model and create this new one.
c) Point 3.b) could introduce some callbacks that would help with autoscaling which would not only be very useful but also pretty cool IMHO.
d) Naturally because it's not threadsafe, we actually have to keep more than one instance of each engine which is again something that can be optimized via some timeout, moving windows, fixed limits, and so on.
e) Would be good to have some sort of middleware architecture here that allows one to gain insights into how models are requested.

What do you think? I know this probably has no place to exist in the CORE repo and maybe belongs more in the userland. But maybe parts of it could? I would very much appreciate any feedback that would lead me to the correct path. Cheers!