jupyter / jupyter/notebook

PySpark RDD <foreach> print does not produce any output

Open
#3,771 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
13.3k
Forks
5.8k
Avg merge
6d 11h
Merged PRs (30d)
7

Description

```
import findspark
findspark.init()

import pyspark
sc = pyspark.SparkContext(appName="RDD (no)Print")

sc.textFile("/user/hadoop/input/dummy.txt").foreach(lambda x:print(x))

sc.stop()
```
The above code does not produce any output. The same example works just fine within pyspark console.

Contributor guide

Open the contributing guide

Research direction

The issue provides only a Python reproducer using findspark, pyspark.SparkContext, textFile, and RDD.foreach; no repository file or test is named. First reproduce the difference between a notebook and the pyspark console, then trace how worker output is handled in the notebook environment. Done means the output behavior is explained and a regression test or documented resolution covers the reproducer.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.