Automatically create the spark history log directory
Open
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
If the directory "spark.history.fs.logDirectory" does not exist, the history server will fail to start. Automatically creating this directory can reduce errors.
Similarly, if the spark.eventLog.dir does not exist, the operation will also fail, even if it is written to S3.
Contributor guide
Research direction
Start by locating the history server handling for spark.history.fs.logDirectory and the event-log handling for spark.eventLog.dir, including the S3 case described in the issue. Verify the directory behavior and existing startup tests; done means missing configured directories no longer cause the history server or event-log operation to fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100