holdenk / holdenk/spark-testing-base

A lot of exceptions during cleanup

Open
#307 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

I've noticed that some extra folders like "spark-uuid" in `System.getProperty("java.io.tmpdir")` are created. But, some of them are not deleted. Problem is somewhere here [https://github.com/holdenk/spark-testing-base/blob/79eef40cdab48ee7aca8902754e3c456f569eea6/core/src/main/1.3/scala/com/holdenkarau/spark/testing/Utils.scala#L109](), it returns true, file is not deleted.

I read a bit about java.io.File#delete - seems that file cannot be deleted by two general reasons: permissions and busying by some process. So, since some of them are deleted - looks like permissions is not the reason. It's really annoying to collect megabytes in temp directory.

Spark version 2.2.1, scala version 2.11

Any thoughts here?

Contributor guide

Open the contributing guide

Research direction

Start at core/src/main/1.3/scala/com/holdenkarau/spark/testing/Utils.scala around line 109, then review java.io.File#delete behavior and reproduce the leftover spark-uuid folders in the temporary directory. The issue is resolved when cleanup reliably removes the generated folders without leaving accumulated files behind.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.