S3 rate limiting on realtime server restart

We recently ran into an issue where we performed a fleetwide restart of our realtime servers. They initially caught up on consumption then proceeded to fall behind due to S3 rate limiting on the S3 table directory. This gradually recovered as the servers were no longer rate limited.

We observed the following log messages: https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java#L484

```
[2022-04-14 18:29:20.176066] 2022/04/14 18:29:20.175 WARN [PinotLLCRealtimeSegmentManager] [grizzly-http-server-59] Caught exception while deleting temporary segment files for segment: table1__37__4760__20220414T1717Z
[2022-04-14 18:29:20.176125] java.io.IOException: software.amazon.awssdk.services.s3.model.S3Exception: Please reduce your request rate. (Service: S3, Status Code: 503, Request ID: redacted, Extended Request ID: redacted)
```

I'm wondering if allowing the table segment tmp dir be configurable would help here since having a separate s3 prefix would allow tmp files to have a separate rate limit. Not entirely clear to me if the contention was due to all the realtime servers hitting this particular listFiles call or if others were happening async.

Start at PinotLLCRealtimeSegmentManager.java around the linked log statement and trace the temporary segment directory operations during realtime server restart. Check whether concurrent listFiles and cleanup requests share the S3 table directory and whether a configurable temporary prefix would isolate them. Done means establishing the contention path and documenting or implementing a concrete mitigation with validation against the observed S3 503 behavior.

Written by the indexing model from the issue text.