holdenk / holdenk/spark-testing-base

Temporary warehouse on local is not a valid hdfs path (SharedSparkContext)

Open
#267 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
357
PR merge metrics
No merged PRs in 30d

Description

If a given test class is using SharedSparkContext and the main application points to an hdfs cluster by default, the application tries to initialize the temporary warehouse in a local directory. Since the local directory is not a valid hdfs path, the test fails. Warehouse should point to file://[whatever] instead of something like C://[whatever]

Contributor guide

Open the contributing guide

Research direction

Start by tracing SharedSparkContext's temporary warehouse initialization and inspect the tests that use it. Confirm how the application resolves a local warehouse when the default points to HDFS, then verify affected tests pass with a file:// warehouse path.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, scala, spark
Domain
data-engineering, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.