Can not version or assign to parent AutoML forecasting (TFT) model when training it from custom KubeFlow container

I am trying to implement Temporal Fusion Transformer (TFT) forecasting model in Vertex AI pipelines. Unfortunately, it is not possible to use Cloud Pipelines API directly, as it does not provide any specialized operator for this task, except [v1.automl.training_job.AutoMLForecastingTrainingJobRunOp](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-2.14.1/api/v1/automl/training_job.html#v1.automl.training_job.AutoMLForecastingTrainingJobRunOp) which performs generic AutoML (or end-to-end (L2L) model training, according to the docs). Hence, I switched to [google.cloud.aiplatform.TemporalFusionTransformerForecastingTrainingJob](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.TemporalFusionTransformerForecastingTrainingJob#google_cloud_aiplatform_TemporalFusionTransformerForecastingTrainingJob_run) and wrapped the whole model training in KFP component as following:

```python
@component(base_image='gcr.io/ml-pipeline/google-cloud-pipeline-components:2.14.1')
def train_tft_model(
project: str,
location: str,
dataset: Input[VertexDataset],
model: Output[VertexModel],
target_column: str,
time_column: str,
time_series_identifier_column: str,
...
):
from google.cloud import aiplatform
aiplatform.init(project=project, location=location)
tft_job = aiplatform.TemporalFusionTransformerForecastingTrainingJob(
project=project,
location=location,
display_name=display_name,
optimization_objective=optimization_objective,
column_transformations=column_transformations
)
tft_model: aip.Model = tft_job.run(
dataset=aiplatform.TimeSeriesDataset(dataset.metadata.get('resourceName')),
target_column=target_column,
time_column=time_column,
...
)
model.metadata["resourceName"] = tft_model.resource_name
```

In the pipeline for this operator I assigned all the relevant parameters, including
```python
is_default_version=True,
parent_model=models[0].resource_name
```
where `models` is retrieved with `api.Models.list`.

After executing pipeline with this component first time, everything is run successfully, and the model is successfully created and registered in Vertex environment. However, after running it second time, the pipeline fails with the following exception `400 Can not upload AutoML model to a custom training parent model`:

```
ERROR 2024-06-21T18:55:59.214788128Z [resource.labels.taskName: workerpool0-0] /tmp/tmp.8PgEHL9QU3/ephemeral_component.py:6: FutureWarning: Google Cloud Pipeline Components will drop support for Python 3.8 on Oct 1, 2024. To use new versions of the GCPC SDK after that date, you will need to upgrade to Python >= 3.9. See https://devguide.python.org/versions/ for more details.
ERROR 2024-06-21T18:55:59.214798328Z [resource.labels.taskName: workerpool0-0] from google_cloud_pipeline_components.types.artifact_types import VertexDataset
ERROR 2024-06-21T18:55:59.214805162Z [resource.labels.taskName: workerpool0-0] Traceback (most recent call last):
ERROR 2024-06-21T18:55:59.214813018Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
ERROR 2024-06-21T18:55:59.214822676Z [resource.labels.taskName: workerpool0-0] return callable_(*args, **kwargs)
ERROR 2024-06-21T18:55:59.214830049Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 1181, in __call__
ERROR 2024-06-21T18:55:59.214837343Z [resource.labels.taskName: workerpool0-0] return _end_unary_response_blocking(state, call, False, None)
ERROR 2024-06-21T18:55:59.214848027Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
ERROR 2024-06-21T18:55:59.214855172Z [resource.labels.taskName: workerpool0-0] raise _InactiveRpcError(state) # pytype: disable=not-instantiable
ERROR 2024-06-21T18:55:59.214862Z [resource.labels.taskName: workerpool0-0] grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
ERROR 2024-06-21T18:55:59.214871211Z [resource.labels.taskName: workerpool0-0] status = StatusCode.INVALID_ARGUMENT
ERROR 2024-06-21T18:55:59.214878602Z [resource.labels.taskName: workerpool0-0] details = "Can not upload AutoML model to a custom training parent model."
ERROR 2024-06-21T18:55:59.214885441Z [resource.labels.taskName: workerpool0-0] debug_error_string = "UNKNOWN:Error received from peer ipv4:172.217.22.42:443 {created_time:"2024-06-21T18:55:39.192347873+00:00", grpc_status:3, grpc_message:"Can not upload AutoML model to a custom training parent model."}"
ERROR 2024-06-21T18:55:59.214892913Z [resource.labels.taskName: workerpool0-0] >
ERROR 2024-06-21T18:55:59.214902409Z [resource.labels.taskName: workerpool0-0]
ERROR 2024-06-21T18:55:59.214909997Z [resource.labels.taskName: workerpool0-0] The above exception was the direct cause of the following exception:
ERROR 2024-06-21T18:55:59.214916270Z [resource.labels.taskName: workerpool0-0]
ERROR 2024-06-21T18:55:59.214925950Z [resource.labels.taskName: workerpool0-0] Traceback (most recent call last):
ERROR 2024-06-21T18:55:59.214933690Z [resource.labels.taskName: workerpool0-0] File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
ERROR 2024-06-21T18:55:59.214940647Z [resource.labels.taskName: workerpool0-0] return _run_code(code, main_globals, None,
ERROR 2024-06-21T18:55:59.214951076Z [resource.labels.taskName: workerpool0-0] File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
ERROR 2024-06-21T18:55:59.214957806Z [resource.labels.taskName: workerpool0-0] exec(code, run_globals)
ERROR 2024-06-21T18:55:59.214964793Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/kfp/dsl/executor_main.py", line 109, in
ERROR 2024-06-21T18:55:59.214974057Z [resource.labels.taskName: workerpool0-0] executor_main()
ERROR 2024-06-21T18:55:59.214980882Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/kfp/dsl/executor_main.py", line 101, in executor_main
ERROR 2024-06-21T18:55:59.214987275Z [resource.labels.taskName: workerpool0-0] output_file = executor.execute()
ERROR 2024-06-21T18:55:59.214997218Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/kfp/dsl/executor.py", line 361, in execute
ERROR 2024-06-21T18:55:59.215003953Z [resource.labels.taskName: workerpool0-0] result = self.func(**func_kwargs)
ERROR 2024-06-21T18:55:59.215011268Z [resource.labels.taskName: workerpool0-0] File "/tmp/tmp.8PgEHL9QU3/ephemeral_component.py", line 62, in train_tft_model
ERROR 2024-06-21T18:55:59.215018860Z [resource.labels.taskName: workerpool0-0] tft_model: aip.Model = tft_job.run(
ERROR 2024-06-21T18:55:59.215029935Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/training_jobs.py", line 2137, in run
ERROR 2024-06-21T18:55:59.215036914Z [resource.labels.taskName: workerpool0-0] return self._run(
ERROR 2024-06-21T18:55:59.215043861Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/base.py", line 863, in wrapper
ERROR 2024-06-21T18:55:59.215053923Z [resource.labels.taskName: workerpool0-0] return method(*args, **kwargs)
ERROR 2024-06-21T18:55:59.215061271Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/training_jobs.py", line 2591, in _run
ERROR 2024-06-21T18:55:59.215068489Z [resource.labels.taskName: workerpool0-0] new_model = self._run_job(
ERROR 2024-06-21T18:55:59.215078867Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/training_jobs.py", line 818, in _run_job
ERROR 2024-06-21T18:55:59.215085901Z [resource.labels.taskName: workerpool0-0] training_pipeline = self.api_client.create_training_pipeline(
ERROR 2024-06-21T18:55:59.215092756Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform_v1/services/pipeline_service/client.py", line 1002, in create_training_pipeline
ERROR 2024-06-21T18:55:59.215103880Z [resource.labels.taskName: workerpool0-0] response = rpc(
ERROR 2024-06-21T18:55:59.215111112Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
ERROR 2024-06-21T18:55:59.215117880Z [resource.labels.taskName: workerpool0-0] return wrapped_func(*args, **kwargs)
ERROR 2024-06-21T18:55:59.215128147Z [resource.labels.taskName: workerpool0-0] File "/usr/local/lib/python3.8/dist-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
ERROR 2024-06-21T18:55:59.215144504Z [resource.labels.taskName: workerpool0-0] raise exceptions.from_grpc_error(exc) from exc
ERROR 2024-06-21T18:55:59.215152253Z [resource.labels.taskName: workerpool0-0] google.api_core.exceptions.InvalidArgument: 400 Can not upload AutoML model to a custom training parent model.
```
This is quite strange as the parent model should have been created as AutoML model in the first place after running the pipeline the first time. I can confirm that this approach works perfectly when I use `AutoMLForecastingTrainingJobRunOp` to train and deploy the model. Is it possible to use similar model versioning for the TFT model?