[KEP-3562] Parallelize TrainJob creation using WorkQueue at scale

### What you would like to be added?

In the `OptimizationJob` controller, parallelize child TrainJob creation when spawning batches of trials (when parallelTrials is large, e.g. > 1000)

Adopt a worker-pool pattern similar to batch controllers like JobSet: https://github.com/kubernetes-sigs/jobset/blob/main/pkg/controllers/jobset_controller.go#L765

### Why is this needed?

Currently, `createTrainJobs` iterates over the generated parameter suggestions and invokes `Create` synchronously for each TrainJob.

Using `WorkQueue` increases concurrency.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍