apache / apache/beam

[Bug]: Cannot read partitions metadata table from bigquery

Open
#24,726 1 comment 0 reactions 0 assignees View on GitHub
awaiting triage bug dataflow io P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Hello,
I wanted to read partitions metadata from bigquery table `project_id.dataset_id.INFORMATION_SCHEMA.PARTITIONS` using `ReadFromBigQuery`. Unfortunately, this function raises an error:
```
RuntimeError: apitools.base.py.exceptions.HttpNotFoundError: HttpError accessing : response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Mon, 19 Dec 2022 17:02:59 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '404', 'content-length': '388', '-content-encoding': 'gzip'}>, content <{
"error": {
"code": 404,
"message": "Not found: Dataset [project_id]:`[table_name] was not found in location EU",
"errors": [
{
"message": "Not found: Dataset [project_id]:`[table_name] was not found in location EU",
"domain": "global",
"reason": "notFound"
}
],
"status": "NOT_FOUND"
}
}
> [while running '[11]: FindPreviousPartitionDate/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction']
```

The part of code that causes the error:
```python
"FindPreviousPartitionDate" >> beam.io.ReadFromBigQuery(
query="SELECT * FROM `[project_id].[dataset_id].INFORMATION_SCHEMA.PARTITIONS`",
use_standard_sql=True,
flatten_results=False
)
```

I replaced my actual project id and dataset id with tokens `[project_id]`, `[dataset_id]`.
I've tested it with beam versions `2.36.0` and `2.43.0` using direct and dataflow runners. Also tried running it with different argument values like `method`, or `use_standard_sql`, but it doesn't help.

Do you know if it is possible to read from this table using `ReadFromBigQuery`

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.