Use Kubeflow metadata for metrics collection

/kind feature

**Describe the solution you'd like**
Right now Katib depends on logging the metrics to stdout (see #685).

It would be nice if instead Katib could be configured to use Kubeflow metadata to obtain the metrics.

Here's a strawman for how this might work

1. User adds logging statement to their code to log metrics to metadata with an appropriate set of labels (e.g. experiment & trial)
1. Katib use a selector to match trials to metrics in metadata

It seems natural for folks to instrument their code to log metrics to metadata.

Furthermore, using the metadata SDK to log metrics should mean logging metrics to metadata is no more difficult then logging to stdout.

A side benefit would be that this avoids some of the sideffects of using side cars to fetch logs from stdout (#685)

* Sidecars make it more difficult to determine when a job is completed.
* Logging to metadata its easier to write robust code to ensure that metrics are logged
* Training code gets an ACK from the metadata store and can retry in the event of failure
* In contrast if we rely on training code printing to stdout and being collected asynchronously the training code has no way of knowing whether metrics have been successfully preserved.

/cc @zhenghuiwang @johnugeorge @gaocegege

No files or tests are named. Start by reading issue #685 and the Kubeflow Metadata SDK material, then trace Katib's existing stdout metric collection path. Done would include a configurable metadata-based path that matches trials to logged metrics and replaces the relevant stdout dependency.

Written by the indexing model from the issue text.