Data locality for DoFns
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
In some distributed filesystems, such as HDFS, we should be able to hint to Spark the preferred locations of splits.
Here is an example of how Spark does that for Hadoop RDDs:
https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala#L249
Imported from Jira [BEAM-673](https://issues.apache.org/jira/browse/BEAM-673). Original Jira may contain additional context.
Reported by: amitsela.
Contributor guide
Research direction
Read Spark's linked NewHadoopRDD.scala example and investigate how Beam DoFns and input splits are handled for Spark execution. Determine the entry point for conveying preferred locations from HDFS-like filesystems, with completion meaning those split locations can be used by Spark for data-local execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100