linkedin / linkedin/data-integration-library

Adding text extractor for extracting unstructured output

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
34
Forks
22
PR merge metrics
No merged PRs in 30d

Description

Currently DIL supports many structured format like CSV, Json, Avro and also many compression formats. Unstructured text format is supported only through FileDumpExtractor, which dumps output to HDFS. With FileDumpExtractor, output cannot be passed to any converter. Text Extractor should be supported, which can extract output in any format and pass it to some converter for further ETL rather than directly pushing this to HDFS. This is useful in cases where we want to get some URL output and then apply some custom parsing to get the required output.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating FileDumpExtractor and the existing extractor-to-converter flow. Review how structured extractors pass data onward, then define the Text Extractor behavior for unstructured output and verify that converted results no longer require direct HDFS dumping. Add or update tests covering extraction followed by conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.