[Feature Request] : Application file residing in hdfs
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
Command : spark-submit --class org.apache.spark.deploy.DotnetRunner --deploy-mode cluster --master spark://:7077 hdfs://:9000/user/ContainerAdministrator/NetForSparkDemo/microsoft-spark-2.4.x-0.2.0.jar hdfs://:9000/user/ContainerAdministrator/NetForSparkDemo/NetForSparkDemo.exe
The driver failed with the below exception.
Would there be an approximate timeline for this to be available?
Caused by: java.lang.IllegalArgumentException: Launching Python applications through spark-submit is currently only supported for local files: hdfs://:9000/user/ContainerAdministrator/NetForSparkDemo/NetForSparkDemo.exe
Contributor guide
Research direction
Start by reproducing the supplied spark-submit command and reviewing the reported DotnetRunner exception for HDFS application paths. Trace how application locations are validated and launched; done means the .exe can be supplied from HDFS in cluster mode without the local-files-only failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, hadoop
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100