googleapis / googleapis/google-cloud-go

bigquery: allow nullable tags on non nullable attributes

Open
#2,814 2 comments 0 reactions 1 assignee Claimed by @shollyman View on GitHub
api: bigquery priority: p2 type: feature request
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

We have a pipeline ingesting json data into GCS.
We then load GCS data into BigQuery tables on demand when needed.

We currently maintain a hardcoded schema for each table type, however it seems we are very close from being able to leverage json like tags offered by the library.

Let's take an example.

At a moment `T`, we have the following struct:
```go
type Row struct {
AppID string `json:"app_id" bigquery:"app_id"`
}
```

We start pushing data to GCS and import some of it into BigQuery inferring the table schema from the struct:
```go
schema, _ := bigquery.InferSchema(Row{})
```

So far so good.
Now we introduce a new field at `T+1`:

```go
type Row struct {
AppID string `json:"app_id" bigquery:"app_id"`
NbHits int `json:"nb_hits" bigquery:"nb_hits"`
}
```

We infer the schema from this new schema and update existing tables or delete them (it does not really matter).

The problem now is that we can no longer load data uploaded to GCS at `T` time, because it does not have data for `nb_hits`.

I completely understand that it makes little sense in context of go only to allow a nullable field on an attribute that cannot be `nil`. However in our case, I feel like given we are importing data from other places into BigQuery, it would make a lot of sense allowing a `nullable` type on any attribute of the struct.

If the attribute is cannot be `nil` in the struct, then we could always use the default value for the attribute instead.
Doing so would allow us to leverage the same struct to infer the schema of our tables.

For now, the workaround I see is relaxing all the fields:
```go
chema, _ := bigquery.InferSchema(Row{})
schema = schema.Relax()
```

Of course we would love to keep some of the required fields and ideally keep that definition in the tags.

Last but not least, using the nullable types introduced by the `bigquery` package is not an option for us because the impact on the codebase would be big and introduce more complexity with the need to handle the `nil`.

With JSON marshalling and unmarshalling, when a field is absent the attribute just takes the default value of the attribute type. It would be awesome if your library could adopt the same approach.

I would love to be able to write at `T+1`:
```go
type Row struct {
AppID string `json:"app_id" bigquery:"app_id"`
NbHits int `json:"nb_hits" bigquery:"nb_hits,nullable"`
}
```

I do realize that in this specific case, `nb_hits` being absent or equalling `0` would have a whole different meaning.
But sometimes it is fine. Most of the cases, for `string`s, depending on the business use case, an empty string equals `NULL`. Also in our case, our problem is more about being able to backfill data from GCS having missing fields.

Happy to have your thoughts on the subject.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.