google-deepmind / google-deepmind/geeflow
Dataflow arguments are ignored and pipeline defaults to DirectRunner
- Dominant language
- Python
- Stars
- 132
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
When executing the `geeflow.export_beam_tfds` module with the `--runner=DataflowRunner` and appropriate GCP-related flags, the job still defaults to DirectRunner instead of launching on Google Cloud Dataflow.
## Command
```bash
python -m geeflow.export_beam_tfds \
--config_path=config_advanced_ndvi_sar.py \
--output_dir=$OUTPUT_DIR \
--ee_project=$EE_PROJECT \
--running_mode=cloud \
--tfds_name=$TFDS_NAME \
--splits=train \
--file_format=tfrecord \
-- \
--runner=DataflowRunner \
--project=$GCP_PROJECT \
--region=$REGION \
--temp_location=$TEMP_LOCATION \
--staging_location=$STAGING_LOCATION \
--max_num_workers=50 \
--job_name=satellite-data-extraction
```
## Expected Behavior
The job should be executed using Google Cloud Dataflow as specified by the `--runner=DataflowRunner` argument.
## Actual Behavior
The job executes using the default `DirectRunner`, and the Dataflow-specific options (`--project`, `--region`, etc.) are ignored.
## Environment and Config Context
Relevant config functions are defined and used properly in script with labels set. The `running_mode` is set to `cloud`, and the `tfds_name`, `splits`, and `file_format` are properly specified.
## Logs
Logs can be found in `output.log`.
[output.log](https://github.com/user-attachments/files/20834081/output.log)
Contributor guide
Assessment
This issue has not been assessed yet.