Design for NAS model constructor

/kind feature

Currently, for ENAS we have model constructor inside training container. I think, we can create Model somewhere else. In that case, user doesn't need to create model constructor inside training container.

Few thoughts:

1. As @gaocegege mentioned here: https://github.com/kubeflow/katib/issues/461, we can use ONNX representation and create mode inside Suggestion. In that case, we can extend `GetSuggestionsReply` with additional field: `bytes onnx_model`. Or convert bytes to string, if it is possible.

2. We can use JSON representation for the model (https://www.tensorflow.org/guide/keras/save_and_serialize#architecture-only_saving), but it will not save the weights. For `enas` it doesn't matter, but for other algorithms, maybe, we have to send model to Trials with predefined weights.

3. I think we should create model inside Suggestion, since Trial is just a training job. For NAS algorithms user wants to receive the model in the training job. Also, for example `darts` algorithm mades some modifications with current running model during the training.

/cc @johnugeorge @gaocegege

Start by reviewing the ENAS model-constructor flow and the GetSuggestionsReply, Suggestion, and Trial roles described in this issue, along with the linked discussion in issue 461. Compare the ONNX and JSON approaches, including the treatment of weights and DARTS modifications. Done means there is an agreed design for where models are created and how the resulting model reaches the training job.

Written by the indexing model from the issue text.