googleapis / googleapis/google-cloud-go
aiplatform: ImportDataRequest missing ImportSchemaUri for tabular data
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.6k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 109
Description
I have created the empty dataset (using the Go SDK, but maybe the language doesn't matter), and the next step is to associate the data source with the dataset. When creating the empty dataset, I specified the "metadata schema URI" as `gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml`. The empty dataset has been created with no problem, and it appeared on the vertex AI dashboard.
The data is already uploaded to a bucket on Google Storage. So far so good. However the "import data request" requires the attribute "Import Schema URI" to have a valid value.
Unfortunately, all the available schemas are not for tabular data AFAIK. When listing the content of the bucket `gs://google-cloud-aiplatform/schema/dataset/ioformat` in fact, there's nothing about tabular data.
Am I doing something wrong or is this schema for the IO of tabular data missing?
e.g.
```go
import (
vai "cloud.google.com/go/aiplatform/apiv1beta1"
vaipb "cloud.google.com/go/aiplatform/apiv1beta1/aiplatformpb"
)
// lots of other code
// Create the dataset
// ref: https://pkg.go.dev/cloud.google.com/go/aiplatform/apiv1beta1/aiplatformpb#CreateDatasetRequest
req := &vaipb.CreateDatasetRequest{
// Required. The resource name of the Location to create the Dataset in.
// Format: `projects/{project}/locations/{location}`
Parent: fmt.Sprintf("projects/%s/locations/%s", _vaiProjectID, _vaiLocation),
Dataset: &vaipb.Dataset{
DisplayName: userDataset,
Description: fmt.Sprintf("User %d data", user.ID),
MetadataSchemaUri: "gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml",
},
}
var createDatasetOp *vai.CreateDatasetOperation
if createDatasetOp, err = datasetClient.CreateDataset(ctx, req); err != nil {
return err
}
if dataset, err = createDatasetOp.Wait(ctx); err != nil {
return err
}
// So far OK!
importDataReq := &vaipb.ImportDataRequest{
Name: datasetFullPath,
ImportConfigs: []*vaipb.ImportDataConfig{
{
Source: &vaipb.ImportDataConfig_GcsSource{GcsSource: &vaipb.GcsSource{Uris: []string{fmt.Sprintf("gs://%s/%s", obj.BucketName(), obj.ObjectName())}}},
// BELOW IS THE PROBLEM: WHAT'S THE CORRECT SCHEMA?
ImportSchemaUri: "l",
}},
}
var importDataOp *vai.ImportDataOperation
if importDataOp, err = datasetClient.ImportData(ctx, importDataReq); err != nil {
return err
}
// Error!
```
**Expected behavior**
I should be able to import the tabular data into the dataset.
**Actual behavior**
If I leave the `ImportSchemaUri` attribute empty, I receive this message
```
Required field is not set.\t\nerror details: name = BadRequest field = name import_configs[0].import_schema_uri desc = Invalid Dataset resource name. Required field is not set.
```
If I put a wrong schema (like one of the other ioformat that's not related to tabular data, I receive the message
```
rpc error: code = InvalidArgument desc = List of found errors:\t1.Field: name; Message: Invalid Dataset resource name.\t\nerror details: name = BadRequest field = name desc = Invalid Dataset resource name.
```
Anyway, I do really thing the problem is in the missing schema for the tabular data. Is this correct? Is there a workaround for this?
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.