Request tracing prevents query completion

There are currently a few bugs in the tracing implementation:

1. `TraceContext` accessed concurrently from different threads leading to `ConcurrentModificationException`
```
2021/09/30 17:47:38.875 ERROR [QueryScheduler] [pqr-12] Encountered exception while processing requestId 7 from broker Broker_192.168.0.22_8000
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) ~[?:?]
at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
at org.apache.pinot.core.util.trace.TraceContext$Trace.toJson(TraceContext.java:90) ~[classes/:?]
at org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:189) ~[classes/:?]
at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236) ~[classes/:?]
at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138) ~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [guava-20.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [guava-20.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [guava-20.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```

2. `NullPointerException` in query runner thread pool after the request id has been cleared
```
2021/09/30 17:47:48.858 ERROR [QueryScheduler] [pqr-15] Encountered exception while processing requestId 8 from broker Broker_192.168.0.22_8000
java.lang.NullPointerException: null
at org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:188) ~[classes/:?]
at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236) ~[classes/:?]
at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154) ~[classes/:?]
at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138) ~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [guava-20.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [guava-20.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [guava-20.0.jar:?]
```

3. Tracing errors cause the query to fail (tracing errors should never impact what's being traced, it should simply be recorded that tracing failed without changing behaviour.)

Inspect TraceContext, especially Trace.toJson and getTraceInfo, then follow their calls from ServerQueryExecutorV1Impl.processQuery and QueryScheduler. Reproduce the concurrent-modification and cleared-request-id failures if possible. Done means tracing failures are recorded without causing the query itself to fail.

Written by the indexing model from the issue text.