linkedin / linkedin/brooklin

S3 connector

Open
#615 0 comments 9 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
968
Forks
144
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Similarly to the file connector, ingesting data from S3 would be fantastic.
S3 can emit notifications of new files onto SQS, Kinesis, etc. so it may be beneficial to hook in there.

Essentially, it would be great if Brooklin could be notified of new S3 files and then ingest the actual files, so we can output them onto Kafka.

It may be necessary to differentiate between different types of files
- Plain-text line-by-line
- Single-line JSON objects
- Pretty-printed JSON

Finally, using `import java.util.zip.{ GZIPInputStream, ZipInputStream }`, files could be unarchived on-the-fly.

**Describe the solution you'd like**
Provide the system with an S3 bucket and credentials.
New S3 files will be streamed into data sink (the AWS REST API allows actual streaming of files). Depending on type of file, apply different logic to unarchive/read (see above).
I'd like to have the file streamed into separate Kafka messages depending on the above logic.

For example:

- New file `foo.tar.gz` is written to S3
- Notification is emitted by AWS
- File is streamed into Brooklin
- File is automatically unarchived using `GZIPInputStream`
- File contains `1.json` and `2.json`, which have pretty printed JSON objects inside
- Send each JSON object from each of the file in a separate message onto Kafka

**Describe alternatives you've considered**
- Custom implementation of the above logic using an SQS client and Kafka Streams
- Kafka Connect has an S3 connector but the officially supported one only allows Kafka -> S3, not S3 as a source

**Additional context**
This would be an extremely valuable connector when working with systems that can export their data feeds to S3.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining Brooklin’s existing file connector and how it handles notifications or Kafka output. Clarify whether SQS or Kinesis is the notification entry point, which text and JSON formats must be supported, and how GZIPInputStream or ZipInputStream applies. Done means a defined S3-to-Kafka connector with agreed file parsing, decompression, credentials, and coverage for the stated example.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, kafka
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.