BigQueryBatchFileLoads dynamic destination
Open
bug
documentation
gcp
good first issue
io
P3
python
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I am trying to use BigQueryBatchFileLoads to upload data from pubsub. It works fine for a single table, but when I try to use a dynamic destination such as
destination=lambda elem: "my_project:my_dataset." **** elem["sensor_key"],
it just makes a new table for each time the triggering_frequency procs. I know it makes temporary tables before loading it all into one, but it never loads them. It just creates more and more tables.
Imported from Jira [BEAM-9917](https://issues.apache.org/jira/browse/BEAM-9917). Original Jira may contain additional context.
Reported by: tordaut.
Contributor guide
Assessment
This issue has not been assessed yet.