[Bug]: Can't preform full integration test on a streaming pipeline
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
**Use Case:**
Test a streaming pipeline end-to-end by passing information to the production pipeline without any code manipulation in the pipeline
**Background information:**
- in order to assert that the pipeline passed the integration test, we pass a dict of extra options with the key on_success_matcher that contains the verifies (which in this case is the `state_verifies` and `pubsub_msg_verifier`)
- The extra options dict gets passed to the `get_full_options_as_args` method which pickles the matchers.
- In the scenario of a TestPipeline the class has the test_append_verifier_in_extra_opt method which is able to "unpickle" the matchers and appends them to the pipeline arguments
- In the Github example (which can be found under [here](https://github.com/apache/beam/blob/ba38a4d6e5501c3e66163e03c992f00c97db5f5a/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py)) the extra options are passed to the streaming count pipeline.

**Issues:**
- The issue I've encountered is that the Pipeline class does not have access to the `test_append_verifier_in_extra_opt` method and therefore does not possess the mechanics required to unpickle the matchers -> means the pipeline does not assert anything as the matchers are appended to the pipeline options while still pickled and therefore are unparsable by the pipeline.

### Issue Priority
Priority: 2
### Issue Component
Component: testing
Contributor guide
Assessment
This issue has not been assessed yet.