How to handle AstroPy changes breaking affiliated package dependencies?

When testing astropy 6, I noticed that it's possible to get a (partly) broken installation by updating astropy while an older, incompatible gwcs version <0.18 is still installed.

It seems that gwcs versions <0.18 don't work with astropy 6, because they depend on functionality that was later split into the stand-alone `asdf-astropy` package, which they don't know about. This causes asdf serialization of gwcs objects to fail. While it's the responsibility of gwcs to specify what astropy versions are compatible, it obviously cannot know about all future changes upstream, nor can the dependency constraints of already-published gwcs versions be modified **reliably** after the fact (because the dependency solver can always choose an older build, before the extra constraint was added, unless old builds are removed from the package repo, which is a Bad Idea for other reasons). Thus, since astropy does not depend on gwcs in turn, there is nothing to prevent the broken combination.

When building the astropy conda package, this particular issue could be avoided by adding `"run_constrained: gwcs >=0.18"` to the recipe, which would disallow installation alongside incompatible gwcs versions, without actually making gwcs into a dependency. However, I'm not sure whether a similar mechanism exists for pip -- and it's too late now anyway, because 6.0 was already published on PyPI yesterday. So this would only be a partial solution.

This is not the end of the World, because users doing a typical new install should get a working combination of packages, but it would be better if the breakage could not occur when updating `astropy` or including additional packages that could hold gwcs back.

I'm therefore wondering whether affiliated/dependent packages should perhaps be using constraints such as `astropy >=5.0,<6.0` (as conda does for python & numpy), rather than leaving the maximum version wide open, in anticipation of such future breakage? Or maybe some of you have other insights regarding best practices for avoiding dependency hell? I might be making a meal out of a very general problem without a general solution, but it's at least good to be aware of the issue, in order to tell users what to update when they run into it.

(I suppose for DRAGONS, we'll just require `gwcs >=0.18` when we publish v3.2 next year and then the issue will mostly go away for us.)

cc: @saimn, @pllim, @mwcraig, @astrofrog, @nden, @olebole

Start by reviewing the astropy 6 and gwcs versions below 0.18 failure scenario described here, then compare the conda run_constrained approach with what pip supports. Done means documenting a decided project policy or recommendation for preventing incompatible dependency combinations, including how users should recover from an already-broken update.

Written by the indexing model from the issue text.