Esri / Esri/spatial-framework-for-hadoop

Split Enclosed JSON

Open
#71 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
376
Forks
159
PR merge metrics
No merged PRs in 30d

Description

Currently, Enclosed JSON - i.e. a file of complete valid Esri REST JSON - is treated as non-splittable; in other words, if used as an input to a MapReduce job, the whole file will be processed by a single Mapper.

It would be possible to skip the opening enclosing brace and the metadata section outside the "features" section, and otherwise the RecordReader could be similar to `UnenclosedJsonRecordReader`. In fact, if the Enclosed JSON is restricted to the convention of the metadata preceding the "features", likely only small differences from `UnenclosedJsonRecordReader` would suffice.

Collaborators do not have information as to how much Enclosed JSON is used as MapReduce input. The idea will be set aside, and can potentially pursued if use cases are presented.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing UnenclosedJsonRecordReader and the MapReduce RecordReader behavior for enclosed Esri REST JSON. Before implementation, establish a concrete use case for enclosed JSON as MapReduce input; done would mean a supported splittable reader for the stated features format with appropriate validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.