"Stream is closed" error, but program finishes executing correctly
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
I'm running a .NET Spark app for batch processing with a selection of GitHub projects data. My program runs as expected up through making a Spark Sql call:
```csharp
// ...Code creating spark session, reading into data frame, doing some sorting...
// Then move on to creating UDF, SQL call:
spark.Udf().Register("MyUDF", (date) => DateTest(date));
cleanedProjects.CreateOrReplaceTempView("dateView");
DataFrame dateDf = spark.Sql("SELECT *, MyUDF(dateView.updated_at) AS datebefore FROM dateView");
```
After the Sql query, I perform a Filter() and Drop() for some final processing:
```csharp
DataFrame filteredDates = dateDf.Filter(dateDf["datebefore"] == true);
filteredDates.Show();
filteredDates.Drop("datebefore").Show();
```
After I Show() each of these modified DataFrames, I get the error: `ProcessStream() failed with exception: System.ArgumentException: The stream is closed.` However, even with that error, both calls to Show() do execute successfully and see the correct output.
Why might I be receiving this error (twice)? Since I get the correct output, the error doesn't seem to be affecting anything?
Contributor guide
Assessment
This issue has not been assessed yet.