GoogleCloudPlatform / GoogleCloudPlatform/deploymentmanager-samples

BigQuery log export

Open
#173 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jinja
Stars
951
Forks
700
PR merge metrics
No merged PRs in 30d

Description

Hi,
I'm trying to export some logs to bigquery with the following configuration:

```
- name: BQ_LOG-bigquery-dataset
type: gcp-types/bigquery-v2:datasets
properties:
access:
- role: OWNER
specialGroup: projectOwners
datasetReference:
datasetId: BQ_LOG
projectId: my-project
description: Dataset BigQuery Logs Export
friendlyName: Dataset BigQuery Logs Export
location: EU
- name: log-sink-BQ_LOG-bigquery
type: gcp-types/logging-v2:projects.sinks
properties:
destination: bigquery.googleapis.com/projects/my-project/datasets/$(ref.BQ_LOG-bigquery-dataset.datasetReference.datasetId)
filter: resource.type="bigquery_resource"
project: my-project
sink: log-sink-BQ_LOG-bigquery
uniqueWriterIdentity: true
- name: BQ_LOG-bigquery-dataset-set-sink-iam
action: gcp-types/bigquery-v2:bigquery.datasets.patch
properties:
accesses:
- role: OWNER
specialGroup: projectOwners
- role: WRITER
userByEmail: $(ref.log-sink-BQ_LOG-bigquery.writerIdentity)
datasetId: $(ref.BQ_LOG-bigquery-dataset.datasetReference.datasetId)
projectId: my-project
```

What happen is:
- the dataset BQ_LOG is created
- the sink log-sink-BQ_LOG-bigquery is created
- I've no evidence in Stackdriver Logging about bigquery.datasets.patch action
- the dataset BQ_LOG at the end has only the **projectOwners:OWNER** access
- the DM finish successfully, but as I've said the deployment doesn't make what I want

I've found that the issue is _accesses_ and the right config is _access_ , but no error is fired!

With the correct configuration instead:
- the following error is fired: Email address contains invalid user name
- the _translated_ configuration for BQ_LOG-bigquery-dataset-set-sink-iam resource has **userByEmail: serviceAccount:p770900990982-803178@gcp-sa-logging.iam.gserviceaccount.com**

I don't know how to take a substring of a referenced value.
Do you know how can I achieve the target?

Thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.