Improve mysql write performance and stability when offloading

# Summary

We enabled node status offload and workflows archiving, and we have observed some performance and stability issues.

- there are many slow queries of mysql when running thousands of workflows parallelly, most of which is insert into argo_workflows table, and daily archived workflows cleaning-up may take very long time
- if mysql temporarily unavailable, then many workflows will become Error, this is because controller can not hydrate workflow or dehydrate workflow with mysql
- there may be workflows failed with message "Dead lock found when trying to get lock; trying restarting transaction"

## Use Cases

When running a large cluster, kube-apiserver may be the performance bottleneck, we need offload and archive to improve kube-apiserver performance. currently, it needs some optimisation and improvement to make it stable in a production eviroment.

## Some thoughts

- add appropriate indexes to archive_workflows and argo_workflows table
- compress nodes statues or workflow before save it to mysql
- add workflow to work queue again when Hydrate failed, and don't Dehydrate workflows when mysql is unavailable
- considering using redis or s3 to save node status or archived workflows

Start by tracing the controller's workflow hydration and dehydration paths and the MySQL operations on the argo_workflows and archive_workflows tables. Reproduce the reported slow inserts, cleanup delays, unavailable-MySQL failures, and deadlocks, then define and verify a focused solution for the selected performance and stability problems.

Written by the indexing model from the issue text.