GoogleCloudPlatform / GoogleCloudPlatform/training-data-analyst

"Serverless Data Processing with Dataflow - Writing an ETL Pipeline using Apache Beam and Cloud Dataflow (Python)" job fails because of short schema

Open
#2,541 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
8.6k
Forks
6.1k
Avg merge
4h 44m
Merged PRs (30d)
2

Description

When following the instructions of https://www.cloudskillsboost.google/course_sessions/11591045/labs/433174 (part of `09 Serverless Data Processing with Dataflow: Develop Pipelines`, `[Data Engineer Learning Path](https://www.cloudskillsboost.google/paths/16) > [Serverless Data Processing with Dataflow: Develop Pipelines](https://www.cloudskillsboost.google/paths/16/course_templates/229)
> Beam Concepts Review)

`Task 5. Write to a sink` cites a too short schema:
```
table_schema = {
"fields": [
{
"name": "name",
"type": "STRING"
},
{
"name": "id",
"type": "INTEGER",
"mode": "REQUIRED"
},
{
"name": "balance",
"type": "FLOAT",
"mode": "REQUIRED"
}
]
}
```

However if someone digs deep can see https://github.com/GoogleCloudPlatform/training-data-analyst/blob/989aa2d423f17647b20e2e02382b5d0f7b467193/quests/dataflow_python/batch_event_generator.py#L47 `log_fields = ["ip", "user_id", "lat", "lng", "timestamp", "http_request", "http_response", "num_bytes", "user_agent"]` and consequently the solution file has

```
table_schema = {
"fields": [
{
"name": "ip",
"type": "STRING"
},
{
"name": "user_id",
"type": "STRING"
},
{
"name": "lat",
"type": "FLOAT"
},
{
"name": "lng",
"type": "FLOAT"
},
{
"name": "timestamp",
"type": "STRING"
},
{
"name": "http_request",
"type": "STRING"
},
{
"name": "http_response",
"type": "INTEGER"
},
{
"name": "num_bytes",
"type": "INTEGER"
},
{
"name": "user_agent",
"type": "STRING"
}
]
}
```

however without peeking into the solution the job fails. The instructions could be updates for better student success.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.