[Discussion] What if broken Client connections didn't release Futures at all?

@graingert and I were discussing https://github.com/dask/distributed/issues/5667, and he raised a good question: why should we interpret a broken connection to a Client as an indicator that the Client no longer cares about its futures?

Currently, when the connection to a client breaks, the scheduler releases all its futures. This is clearly too aggressive—small network blips can cause you to lose all your work (https://github.com/dask/distributed/issues/5667). There's been discussion around adding a "grace period", where disconnected clients have some amount of time to reconnect before we release their futures.

But what if we went to the other extreme, and said that unless a client explicitly _tells_ us it's closing, we assume it still cares about its futures, regardless of whether it's around to see them? And whenever it reconnects (if it does), all state updates are synced?

This would of course solve #5667, but also would open the door to many other use-cases, which right now you'd have to solve with [other libraries](https://github.com/eriknw/afar) or [Jupyter notebooks running on a server](https://coiled.io/blog/why-coiled-passed-on-jupyter-notebooks/):
* You live in a rural area with a bad ISP; after submitting a large job to a cluster, your connection goes down for 30min
* You submit some work from your laptop at a café, but they're closing early. You'd like to go to a different café, but if you close your laptop, your tasks will be cancelled. Too late to learn how to change some `client-disconnect-grace-period` on the fly on the scheduler!
* You want to submit a large job on Friday afternoon that ends in a `to_parquet` writing to S3. You don't need the results of the tasks; you're just running them for their side effects. You want to fire-and-forget the tasks without having to learn about [`fire_and_forget`](https://docs.dask.org/en/stable/futures.html#fire-and-forget) because it's a Friday afternoon and nobody wants googling and reading documentation to be the thing between them and their weekend.

I have a feeling that the current client-disconnect-releases-Futures paradigm makes sense in an environment where many users are sharing one cluster. If someone disconnects, it's more neighborly to not let their Futures hang around forever.
But I'm not sure whether multi-user clusters are really a common use-case of distributed anymore? With so many tools to create clusters on-demand (dask-cloudprovider, dask-gateway, pangeo, Coiled, Saturn Cloud, etc.), ephemeral, single-user clusters seem to me like the more typical pattern.

And when you have an ephemeral, single-user cluster, it's probably fine/a good thing if your futures stick around unless explicitly dropped:
* If your local session crashed and you reconnect to the same cluster, and you do want to start from scratch, you can always `client.restart()`
* If you have a scheduler `idle-timeout` set, and your client disconnects for a long time, the cluster _will_ shut itself down one all your futures are complete and the idle timeout has passed. So if you were fire-and-forgetting, that's perfect. If your internet went down all day and you did intend to come back and retrieve those results, the cluster would probably have finished and shut itself down by then. But you probably would prefer that, since you set the idle timeout.

Certainly, the implementation of this versus a grace period can be effectively the same (especially if you can say `grace-period: inf`). But maybe rethinking the semantics of disconnection vs the implementation will be helpful as we figure out what to do here.

cc @fjetter @jcrist @crusaderky @jrbourbeau