holdenk / holdenk/spark-testing-base

how to test foreachRDD in python

Open
#63 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

how would one go about testing functions written for foreachRDD in python?

for output operation, i would like to write to kafka.

I'm thinking in order to do this, i should either write to kafka or a mock kafka operation, which can hold output written from the foreachRDD function. Also, in spark-testing-base, there should be a "run_func" like method to test functions used in foreachRDD.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; its entry point is testing Python functions used by foreachRDD. Start by clarifying whether Kafka or a mock Kafka sink is required and what a run_func-style helper should accept. Done should include a documented, repeatable way to test foreachRDD output and verify the expected Kafka interaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, python
Domain
stream-processing, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.