google-deepmind / google-deepmind/geeflow

Dataflow arguments are ignored and pipeline defaults to DirectRunner

Open
#1 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
132
Forks
17
PR merge metrics
No merged PRs in 30d

Description

## Summary

When executing the `geeflow.export_beam_tfds` module with the `--runner=DataflowRunner` and appropriate GCP-related flags, the job still defaults to DirectRunner instead of launching on Google Cloud Dataflow.

## Command

```bash
python -m geeflow.export_beam_tfds \
--config_path=config_advanced_ndvi_sar.py \
--output_dir=$OUTPUT_DIR \
--ee_project=$EE_PROJECT \
--running_mode=cloud \
--tfds_name=$TFDS_NAME \
--splits=train \
--file_format=tfrecord \
-- \
--runner=DataflowRunner \
--project=$GCP_PROJECT \
--region=$REGION \
--temp_location=$TEMP_LOCATION \
--staging_location=$STAGING_LOCATION \
--max_num_workers=50 \
--job_name=satellite-data-extraction
```

## Expected Behavior

The job should be executed using Google Cloud Dataflow as specified by the `--runner=DataflowRunner` argument.

## Actual Behavior

The job executes using the default `DirectRunner`, and the Dataflow-specific options (`--project`, `--region`, etc.) are ignored.

## Environment and Config Context

Relevant config functions are defined and used properly in script with labels set. The `running_mode` is set to `cloud`, and the `tfds_name`, `splits`, and `file_format` are properly specified.

## Logs
Logs can be found in `output.log`.

[output.log](https://github.com/user-attachments/files/20834081/output.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.