Pyspark maintenance issues

This is a summary issue for the current pyspark module which has some serious issues, many of which go back to 2022.

Issues:
- #8398
- #8399
- #8422
- #8318
- #8337
- #8109
- https://github.com/dmlc/xgboost/issues/8354
- #9626
- https://github.com/dmlc/xgboost/issues/8608

In my opinion the most serious issue is that the pyspark functionality was never documented. A small example is provided but this covers very little of its features. The only way to learn how to use it and which parameters it supports by going through the tests.

The tests are inefficiently written both in terms of lines of code and they take up a disproportionately large amount of CI time, costing developer productivity and money from xgboost's limited CI budget.

In general I don't think this module meets the quality standard of the rest of xgboost's language extensions. I would recommend at the least accepting no new feature development until some of the above is resolved. I think there is a case for removing pyspark entirely considering that it is undocumented and unmaintained. It could be returned if these issues are substantially resolved.

@dmlc/xgboost-committer
@wbo4958 @WeichenXu123

@medb tagging you as you have an active PR on this (which we greatly appreciate).

Start by reviewing the linked issues (#8398, #8399, #8422, #8318, #8337, #8109, #9626, #8354, and #8608), the existing PySpark example, and the PySpark tests. The issue calls for substantially improving documentation, addressing inefficient tests and CI cost, and resolving broader maintenance concerns, but it does not define a bounded completion target.

Written by the indexing model from the issue text.