Formalize `mode` / safety guarantees for Zarr

What is your issue?

It sounds like we're coalescing on when it's safe to write concurrently:

• mode="r+" is safe to write concurrently to different parts of a dataset
• mode="a" isn't safe, because it changes the shape of an array, for example extending a dimension

What are the existing operations that aren't consistent with this?

• Is concurrently writing additional variables safe? Or it requires updating the centralized consolidated metadata? Currently that requires mode="a", which is overly conservative based on the above rules assuming it is safe — we can liberalize to allow with mode="r+".
• https://github.com/pydata/xarray/issues/8371, but that's a bug — edit: or possibly an artifact of writing concurrently to overlapping chunks with a single to_zarr call. We could at least restrict non-aligned writes to mode="a", so it wasn't possible to hit this mistakenly while writing to different parts of a dataset.
• Writing the same values to the same chunks concurrently isn't safe at the moment — we'll get an "Stale file handle" error if two processes write to the same location at the same time. I'm not sure if that's possible to allow; possibly it requires work on the Zarr side. If it were possible, we wouldn't have to be as careful about ensuring that each process has mutually exclusive chunks to write. (lower priority)

No implementation files or tests are named. Start by reviewing current xarray/Zarr handling of mode="r+" and mode="a", then read issue 8371 and its linked discussion about overlapping chunks. Done should include an agreed safety model for concurrent writes and corresponding implementation or documentation changes.