holdenk / holdenk/spark-testing-base
how to test foreachRDD in python
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
how would one go about testing functions written for foreachRDD in python?
for output operation, i would like to write to kafka.
I'm thinking in order to do this, i should either write to kafka or a mock kafka operation, which can hold output written from the foreachRDD function. Also, in spark-testing-base, there should be a "run_func" like method to test functions used in foreachRDD.
Contributor guide
Research direction
The issue names no files or tests; its entry point is testing Python functions used by foreachRDD. Start by clarifying whether Kafka or a mock Kafka sink is required and what a run_func-style helper should accept. Done should include a documented, repeatable way to test foreachRDD output and verify the expected Kafka interaction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, python
- Domain
- stream-processing, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100