holdenk / holdenk/spark-testing-base
A lot of exceptions during cleanup
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
I've noticed that some extra folders like "spark-uuid" in `System.getProperty("java.io.tmpdir")` are created. But, some of them are not deleted. Problem is somewhere here [https://github.com/holdenk/spark-testing-base/blob/79eef40cdab48ee7aca8902754e3c456f569eea6/core/src/main/1.3/scala/com/holdenkarau/spark/testing/Utils.scala#L109](), it returns true, file is not deleted.
I read a bit about java.io.File#delete - seems that file cannot be deleted by two general reasons: permissions and busying by some process. So, since some of them are deleted - looks like permissions is not the reason. It's really annoying to collect megabytes in temp directory.
Spark version 2.2.1, scala version 2.11
Any thoughts here?
Contributor guide
Research direction
Start at core/src/main/1.3/scala/com/holdenkarau/spark/testing/Utils.scala around line 109, then review java.io.File#delete behavior and reproduce the leftover spark-uuid folders in the temporary directory. The issue is resolved when cleanup reliably removes the generated folders without leaving accumulated files behind.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100