An EEL sink for Flume

The experimental _kite data set sink_ exist for _Flume 1.6.0_ which looks on the face of it has the capability of ingesting directly into Hive tables.
- See the following for a comprehensive list of flume sinks: https://flume.apache.org/FlumeUserGuide.html#flume-sinks
- See the following for a comprehensive list of flume sources: https://flume.apache.org/FlumeUserGuide.html#flume-sources
- The canonical in-memory format of a flume event is simply called Event (see https://flume.apache.org/releases/content/1.2.0/apidocs/org/apache/flume/Event.html) - it consists of a _body_ and _headers_ - note for the AVRO source this is marshalled across the wire as AVRO binary.

Headers are usually used for content based routing and multiplexing an event to different sinks.
- When writing the custom EEL sink one takes an event of the channel (queue), interprets the headers if necessary, transforms the body (payload) into an EEL frame so that it can be passed directly into an EEL sink.

See https://flume.apache.org/FlumeUserGuide.html#kite-dataset-sink
- Now my guess without looking at the source code is that the _Kite Dataset Sink_ extracts the byte stream from the _body_ and deserialises it to a AVRO _GenericRecord_ which in turn can be passed directly into the Kite write API.

I think for EEL we should do something similar:

1) Deserialise the payload to a _GenericRecord_
2) Transform the _GenericRecord_ to an EEL frame - note from each _GenericRecord_ you can ascertain the AVRO schema so it should be trivial to convert to a Frame schema.
3) Pass the frame to the EEL sink.
- Client side you can send events via the AVRO or Thrift RPC client - see https://flume.apache.org/FlumeDeveloperGuide.html#rpc-client-interface

There are various options for batching up events and sending securely over SSL - you could even send via Kafka to a Flume Kafka Source

The issue describes adding a new sink for Apache Flume to write data to EEL. Start by examining the existing Kite Dataset Sink in Flume's source code to understand how it deserializes events to GenericRecord. Then look at the EEL SDK's frame API and sink interface. The work involves implementing a custom Flume sink that transforms Flume Event bodies into EEL frames, handling AVRO or Thrift serialization. Testing will require setting up a Flume agent with the new sink.

Escrito por el modelo de indexación a partir del texto del issue.