51zero / 51zero/eel-sdk

An EEL sink for Flume

Open
#168 4 comments 0 reactions 1 assignee Claimed by @hannesmiller View on GitHub
enhancement help wanted
Dominant language
Scala
Stars
147
Forks
32
PR merge metrics
No merged PRs in 30d

Description

The experimental _kite data set sink_ exist for _Flume 1.6.0_ which looks on the face of it has the capability of ingesting directly into Hive tables.
- See the following for a comprehensive list of flume sinks: https://flume.apache.org/FlumeUserGuide.html#flume-sinks
- See the following for a comprehensive list of flume sources: https://flume.apache.org/FlumeUserGuide.html#flume-sources
- The canonical in-memory format of a flume event is simply called Event (see https://flume.apache.org/releases/content/1.2.0/apidocs/org/apache/flume/Event.html) - it consists of a _body_ and _headers_ - note for the AVRO source this is marshalled across the wire as AVRO binary.

Headers are usually used for content based routing and multiplexing an event to different sinks.
- When writing the custom EEL sink one takes an event of the channel (queue), interprets the headers if necessary, transforms the body (payload) into an EEL frame so that it can be passed directly into an EEL sink.

See https://flume.apache.org/FlumeUserGuide.html#kite-dataset-sink
- Now my guess without looking at the source code is that the _Kite Dataset Sink_ extracts the byte stream from the _body_ and deserialises it to a AVRO _GenericRecord_ which in turn can be passed directly into the Kite write API.

I think for EEL we should do something similar:

1) Deserialise the payload to a _GenericRecord_
2) Transform the _GenericRecord_ to an EEL frame - note from each _GenericRecord_ you can ascertain the AVRO schema so it should be trivial to convert to a Frame schema.
3) Pass the frame to the EEL sink.
- Client side you can send events via the AVRO or Thrift RPC client - see https://flume.apache.org/FlumeDeveloperGuide.html#rpc-client-interface

There are various options for batching up events and sending securely over SSL - you could even send via Kafka to a Flume Kafka Source

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue describes adding a new sink for Apache Flume to write data to EEL. Start by examining the existing Kite Dataset Sink in Flume's source code to understand how it deserializes events to GenericRecord. Then look at the EEL SDK's frame API and sink interface. The work involves implementing a custom Flume sink that transforms Flume Event bodies into EEL frames, handling AVRO or Thrift serialization. Testing will require setting up a Flume agent with the new sink.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, kafka, scala
Domain
backend, data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.