locationtech / locationtech/geomesa
Tools - Support distributed ingest of zip files
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Description
Tools - Support distributed ingest of zip files
Hadoop 2.8 ships with commons-compress 1.4.1, which doesn't support reading zip files from anything other than a java.io.File. We could potentially shade in a later version...
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2582
Key: GEOMESA-2582
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Tue, 12 Mar 2019 17:23:21 -0400
Updated: Thu, 14 Mar 2019 19:40:50 -0400
Affects Version(s): 2.3.0
Comments:
557058:48912df4-0004-4f90-9915-db26684ec006 (Thu, 14 Mar 2019 19:40:50 -0400):
Hmm... I have no qualms about shading a later version...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Tools distributed-ingest path and checking how Hadoop 2.8's bundled commons-compress 1.4.1 handles ZIP input. Evaluate the proposed use of a shaded newer commons-compress version. Done means ZIP files can be ingested through the distributed path instead of requiring java.io.File, with coverage for that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100