retry after context is timeout?

138 [2021/10/25 17:15:34.011 +08:00] [WARN] [mpp.go:393] ["other error"] [txnStartTS=428645617953931268] [storeAddr=10.4.131.219:3930] [error="other error for mpp stream: DB::Exception: Exchange receiver meet error : DB::Exception: tunnel65+69 is timeout"]
139 [2021/10/25 17:15:34.011 +08:00] [WARN] [mpp.go:334] ["establish mpp connection meet error, and retrying"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=91]
140 [2021/10/25 17:15:34.011 +08:00] [WARN] [mpp.go:334] ["establish mpp connection meet error, and retrying"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=81]
141 [2021/10/25 17:15:34.011 +08:00] [WARN] [mpp.go:334] ["establish mpp connection meet error, and retrying"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=100]
142 [2021/10/25 17:15:34.011 +08:00] [WARN] [mpp.go:334] ["establish mpp connection meet error, and retrying"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=110]
143 [2021/10/25 17:15:34.012 +08:00] [WARN] [mpp.go:334] ["establish mpp connection meet error, and retrying"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=96]
144 [2021/10/25 17:15:34.012 +08:00] [WARN] [mpp.go:337] ["establish mpp connection meet error and cannot retry"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=81]
145 [2021/10/25 17:15:34.012 +08:00] [WARN] [mpp.go:334] ["establish mpp connection meet error, and retrying"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=86]
146 [2021/10/25 17:15:34.012 +08:00] [WARN] [mpp.go:334] ["establish mpp connection meet error, and retrying"] [error="rpc error: code = Canceled desc = context canceled"] [timestamp=428645617953931268] [task=126]
[428645617953931268.log](https://github.com/pingcap/tics/files/7452166/428645617953931268.log)
...

the log shows that too rpc errors are resulted from failed grpc context timeout about 1min (one of tasks is blocked and returns timeout error after about 1min, and others timeout in the context). So why need to retry on the fact that context is timeout?

Start with mpp.go around lines 334, 337, and 393, then compare the context-cancellation retry path with the preceding MPP stream timeout in the attached log. Trace how timeout and cancellation errors are classified before retrying. Done means the retry decision for this timeout scenario is explained and the observed behavior is addressed.