Cut back the default intersphinx mapping

Actually I'm not sure why we provide the intersphinx mapping in here for so many optional dependencies but missed some others (e.g. why `h5py` but not `pandas`)?

I suggest to only keep python, numpy and astropy here, and the rest can go into `docs/conf.py` for the package repositories. If we keep it however, consider adding more mappings.

Inspect the default intersphinx mapping in the repository and compare it with the package repositories' docs/conf.py files. Confirm which mappings should remain for Python, NumPy, and Astropy, then verify that optional-dependency mappings are handled in the package documentation configuration.

Written by the indexing model from the issue text.