Esri / Esri/spatial-framework-for-hadoop
Split Enclosed JSON
- Dominant language
- Java
- Stars
- 376
- Forks
- 159
- PR merge metrics
- No merged PRs in 30d
Description
Currently, Enclosed JSON - i.e. a file of complete valid Esri REST JSON - is treated as non-splittable; in other words, if used as an input to a MapReduce job, the whole file will be processed by a single Mapper.
It would be possible to skip the opening enclosing brace and the metadata section outside the "features" section, and otherwise the RecordReader could be similar to `UnenclosedJsonRecordReader`. In fact, if the Enclosed JSON is restricted to the convention of the metadata preceding the "features", likely only small differences from `UnenclosedJsonRecordReader` would suffice.
Collaborators do not have information as to how much Enclosed JSON is used as MapReduce input. The idea will be set aside, and can potentially pursued if use cases are presented.
Contributor guide
Research direction
Start by reviewing UnenclosedJsonRecordReader and the MapReduce RecordReader behavior for enclosed Esri REST JSON. Before implementation, establish a concrete use case for enclosed JSON as MapReduce input; done would mean a supported splittable reader for the stated features format with appropriate validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100