apache / apache/beam

Value Provider functionality broken in python sdk

Open
#20,636 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Whether using a custom function or an IO connector (test with `WriteToBigQuery`), the dataflow job complains that the runtime value provider `get()` is being called from a non-runtime context:

```

import argparse
import logging

import apache_beam as beam
from apache_beam.options.pipeline_options
import PipelineOptions

class ExampleDoF(beam.DoFn):
def __init__(self, value_provider):

self.value_provider = value_provider

def process(self, el):
logging.info(f'el:
{el}')
logging.info(f'value provider: {self.value_provider.get()}')
yield el

class
UserOptions(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
parser.add_value_provider_argument('--value_provider',
type=int)

parser = argparse.ArgumentParser()
known_args, pipeline_args = parser.parse_known_args()
pipeline_options
= PipelineOptions(pipeline_args)
user_options = pipeline_options.view_as(UserOptions)
with beam.Pipeline(options=pipeline_options)
as pipeline:
results = (
pipeline
| beam.Create(['element'])

| beam.ParDo(ExampleDoF(user_options.value_provider))
)

```

Imported from Jira [BEAM-10872](https://issues.apache.org/jira/browse/BEAM-10872). Original Jira may contain additional context.
Reported by: jkarimi.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.