dotnet / dotnet/spark

"Stream is closed" error, but program finishes executing correctly

Open
#249 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

I'm running a .NET Spark app for batch processing with a selection of GitHub projects data. My program runs as expected up through making a Spark Sql call:

```csharp
// ...Code creating spark session, reading into data frame, doing some sorting...
// Then move on to creating UDF, SQL call:
spark.Udf().Register("MyUDF", (date) => DateTest(date));
cleanedProjects.CreateOrReplaceTempView("dateView");
DataFrame dateDf = spark.Sql("SELECT *, MyUDF(dateView.updated_at) AS datebefore FROM dateView");
```

After the Sql query, I perform a Filter() and Drop() for some final processing:
```csharp
DataFrame filteredDates = dateDf.Filter(dateDf["datebefore"] == true);
filteredDates.Show();
filteredDates.Drop("datebefore").Show();
```

After I Show() each of these modified DataFrames, I get the error: `ProcessStream() failed with exception: System.ArgumentException: The stream is closed.` However, even with that error, both calls to Show() do execute successfully and see the correct output.

Why might I be receiving this error (twice)? Since I get the correct output, the error doesn't seem to be affecting anything?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.