holdenk / holdenk/spark-testing-base
Temporary warehouse on local is not a valid hdfs path (SharedSparkContext)
Open
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 357
- PR merge metrics
- No merged PRs in 30d
Description
If a given test class is using SharedSparkContext and the main application points to an hdfs cluster by default, the application tries to initialize the temporary warehouse in a local directory. Since the local directory is not a valid hdfs path, the test fails. Warehouse should point to file://[whatever] instead of something like C://[whatever]
Contributor guide
Research direction
Start by tracing SharedSparkContext's temporary warehouse initialization and inspect the tests that use it. Confirm how the application resolves a local warehouse when the default points to HDFS, then verify affected tests pass with a file:// warehouse path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala, spark
- Domain
- data-engineering, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100