apache / apache/beam

[Bug]: BigQueryIO.read() requires GCS tempLocation when using BigQuery Emulator

Open
#35,777 5 comments 0 reactions 0 assignees View on GitHub
bug io java P2 pinned
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

#### **Description**

When using the **BigQuery emulator** in combination with `BigQueryIO.read()` in **Apache Beam (Java)**, the pipeline fails with the following error when running locally with `DirectRunner`:

```
java.lang.IllegalArgumentException: BigQueryIO.Read needs a GCS temp location to store temp files. This can be set with option --tempLocation.
```

This makes it **impossible to write integration tests** against the emulator using `BigQueryIO.read()` unless a (fake) GCS-compatible path is provided—even though the emulator does not support or require real GCS interaction.

#### **To Reproduce**

Steps to reproduce the behavior:

1. Start BigQuery emulator (e.g., [[goccy/bigquery-emulator](https://github.com/goccy/bigquery-emulator)](https://github.com/goccy/bigquery-emulator))
2. Create a Beam pipeline that reads using `BigQueryIO.readTableRows().from(...)`
3. Set `bigQueryEndpoint` to point to the emulator
4. Omit `tempLocation` (or use a local one)
5. Run with `DirectRunner`

#### **Expected behavior**

The pipeline should be able to run against the emulator.

#### **Environment:**

* Beam version: `2.63.0` (also affects earlier)
* Runner: `DirectRunner`
* Java SDK
* Emulator: `ghcr.io/goccy/bigquery-emulator`

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.