[FEATURE] Make metadata of end-to-end integrity validation stored in the shuffle-server side

By leveraging the PR #2653 , we could end-to-end ensure the data consistency. But, the partition stats stored in the spark driver side, for the normal spark stages, this design runs well. But with the 100000 tasks with 10000 partitions, this will make the Spark driver overload. From the point of cluster spark jobs, some huge jobs will hang when getting the blockManagerIds, that will cost almost 20mins for one reader task, that is unacceptable.

And so we should introduce the extra mechanism to store the metadata in the shuffle-server side.

Start by reviewing PR #2653 and the linked merged PR #2669, then trace how partition stats are currently stored on the Spark driver and used to obtain blockManagerIds. The issue's intended result is to move the end-to-end integrity metadata to the shuffle-server side so large jobs do not overload the driver or leave reader tasks waiting for about 20 minutes.

Written by the indexing model from the issue text.