NiFi - support tar/zip/compressed files

NiFi - support tar/zip/compressed files

Assertion: NiFi supports expanding compressed files using CompressContent, but that creates intermediate flow files and increases disk I/O. If we handle it in our processors, then we don't need to create the intermediate results.

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2585

Key: GEOMESA-2585
Type: Story
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Tue, 19 Mar 2019 12:49:02 -0400
Updated: Wed, 20 Mar 2019 10:48:42 -0400

Comments:

557058:6ae937f2-c11d-4135-b676-800b27bfddf1 (Tue, 19 Mar 2019 15:31:28 -0400):

Doesn't nifi already support this with CompressContent?

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Tue, 19 Mar 2019 15:37:06 -0400):

James Srinivasan It does, but there is some concern that it's slow... I updated the description to reflect that. We haven't really looked into it yet, so there may be better ways to do it in NiFi already... do you know one way or the other?

557058:6ae937f2-c11d-4135-b676-800b27bfddf1 (Wed, 20 Mar 2019 10:37:45 -0400):

The general NiFi approach is to keep processors as simple as possible, and build complexity by chaining them. We use CompressContent then PutGeoMesa for gzipped JSON, but our volumes aren't massive. I'd say if tar/zip/whatever support is already in GeoMesa and you're just talking about exposing some params in the GUI, that's just fine. Implementing decompression separately inside the processor might need more though - maybe ask in NiFi Slack?

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Wed, 20 Mar 2019 10:46:46 -0400):

Yeah, we should probably look deeper into the options already available in NiFi. Andrew Hulbert Austin Heyne

557058:6ae937f2-c11d-4135-b676-800b27bfddf1 (Wed, 20 Mar 2019 10:48:42 -0400):

Well CLI ingest supports some compressed formats (for which I am very grateful), I don't really see any issue if the NiFi processor supports the same formats.

Start by reviewing the NiFi CompressContent approach and GeoMesa's existing CLI ingest support for compressed formats. No source file or test is named, so first locate the NiFi processors and their ingest tests, then confirm which tar/zip formats are in scope. Done means the relevant processor supports the agreed formats without unnecessary intermediate flow files.