How to implement streaming with a custom monad - Why does the effect type have to be part of the API type?

I'm currently trying to create a Servant API including a streaming endpoint with a server implemented using MTL + Conduit .

An API type with a streaming endpoint looks like this (according to the cookbook):
```
type Routes =
"get" :> Get '[JSON] [User]
:<|> "stream" :> StreamGet NewlineFraming JSON (SourceIO User)
```
It becomes immediately obvious that for a regular Get endpoint the API type is independent of the server effect type whereas the streaming endpoint includes the effect type (`IO` in case of `SourceIO`). This is problematic for three reasons:

1. Conceptually, the API type is completely independent of effects. We could theoretically implement the same API type using different effects (for example an `IO` based server in production and an `Identity` based server for testing)

2. When using MTL the source type is parameterized, thus we have to add the parameter also to the API type!
```
type Routes m =
"get" :> Get '[JSON] [User]
:<|> "stream" :> StreamGet NewlineFraming JSON (ConduitT () User m ())
```
At a later point both `Servant.serve` and `Servant.hoistServer` expect a `Proxy (Routes m)`. You can imagine that with MTL the transformer stack type can become very large and typically you never write it out because it is so long. It's supposed to be inferred based on the `runFooT` functions you use to discharge the class constraints.

3. According to the cookbook, `hoistServer` is used to turn a `ServerT api m` into `ServerT api Handler`. If we look at the definition of the `HasServer` instance for the `Stream` combinator, we find that `hoistServerWithContext` has the type
```
hoistServerWithContext :: Proxy (Stream method status framing ctype (Headers h a))
-> Proxy context
-> (forall x. m x -> n x)
-> ServerT (Stream method status framing ctype (Headers h a)) m
-> ServerT (Stream method status framing ctype (Headers h a)) n
```
where `a` would be `ConduitT () User m ()` in our example. It seems that `hoistServer` would only turn `m (ConduitT () User m ())` into `Handler (ConduitT () User m ())` leaving the inner `m` that `Servant.serve` doesn't know how to deal with. Writing an instance for [`ConduitToSourceIO m`](https://hackage.haskell.org/package/servant-conduit-0.15.1/docs/Servant-Conduit.html#t:ConduitToSourceIO) is also not possible because instances have to be global (unless you abuse reflection maybe) and the natural transformation `nt` given to `hoistServer` may depend on local state from the enclosing scope.
This problem also applies if you use a concrete custom monad instead of type classes. So far I haven't been able to figure out how to solve it. Perhaps it's possible to somehow map `hoist nt` over the resulting server.

-------
Does anyone have an idea (or example preferable) how to make a `Stream` endpoint work with MTL?

I have tried solving problem 2 by implementing an immensely complicated concoction of type families that allow me to rewrite an API type replacing every `SourceIO a` with `ConduitT () a m ()`. I can now successfully declare an unparameterized generic API type in terms of `SourceIO a` but provide a generic server in terms of `ConduitT () a m ()`. However, I have been unable to figure out how to write a `hoistServer` function for that. It's probably a dead end.

Another option would be to create completely new `Stream` endpoint types that are independent of effect types and throw out `Servant.API.Streaming` completely. The existing implementation for `HasServer Stream` is
```
instance {-# OVERLAPPING #-}
( MimeRender ctype chunk, ReflectMethod method, KnownNat status,
FramingRender framing, ToSourceIO chunk a,
GetHeaders (Headers h a)
) => HasServer (Stream method status framing ctype (Headers h a)) context where

type ServerT (Stream method status framing ctype (Headers h a)) m = m (Headers h a)

hoistServerWithContext :: Proxy (Stream method status framing ctype (Headers h a))
-> Proxy context
-> (forall x. m x -> n x)
-> ServerT (Stream method status framing ctype (Headers h a)) m
-> ServerT (Stream method status framing ctype (Headers h a)) n
hoistServerWithContext _ _ nt s = nt s
```
If we change `a` to be `(MFunctor s) => s b` then it might be possible to write an implementation of `hoistServerWithContext` that also hoists the inner `m` in `ConduitT () b m ()` but I fear that the type of `hoistServerWithContext` will not allow changing the `a` (unless you can do some trickery with the `ServerT` family). Is it even necessary that we have to have to wrap `ConduitT () b m ()` inside another `m` in `ServerT`?

Start with Servant.API.Streaming, the HasServer Stream instance, and hoistServerWithContext, then review the cookbook's streaming example and ConduitToSourceIO. The issue is seeking a design or example for MTL-compatible streaming whose API type is independent of the effect; done would require an agreed implementation approach and supporting tests or documentation.

Written by the indexing model from the issue text.