[EPIC] Add support for Spark 4.1

This epic tracks support for Apache Spark 4.1([Spark Release 4.1.1](https://spark.apache.org/releases/spark-release-4.1.1.html) ) in Auron (current supported versions: 3.0~3.5).
It is proposed to prioritize Spark 4.1 as the first supported Spark 4.x version to accelerate the Spark 4 compatibility initiative, with the implementation designed for extensibility to enable easy addition of Spark 4.0 support at a later stage if required.

## Sub Tasks
- [x] #1956
- [x] #1961
- [ ] Support Spark 4.1 Unit Tests [To be added]