Kafka Avro Schema Option

It would be great if it would be possible to provide the existing avro schema (from url or local path) option while reading Kafka Avro stream.

Currently running the following code produces the error below

KafkaSource(KafkaSourceConfig("localhost:12345", "consumer"), Set("topic1"), AvroKafkaDeserializer)

Exception in thread "main" java.io.IOException: Not a data file.
at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105)
at org.apache.avro.file.DataFileReader.(DataFileReader.java:97)
at io.eels.component.kafka.AvroKafkaDeserializer$.apply(avro.scala:17)
at io.eels.component.kafka.KafkaSource.schema(KafkaSource.scala:36)
at io.eels.FrameSource.schema$lzycompute(Source.scala:29)
at io.eels.FrameSource.schema(Source.scala:29)
at io.eels.plan.SinkPlan$.apply(SinkPlan.scala:17)
at io.eels.Frame$class.to(Frame.scala:362)
at io.eels.FrameSource.to(Source.scala:23)

Thank you.
Roman.

Look at AvroKafkaDeserializer in avro.scala around line 17, where it tries to read the Avro data. The issue is that the deserializer expects a data file but receives raw Avro records. Investigate how to accept a schema URL or local path, possibly using Avro's Schema.Parser. Check KafkaSource.scala line 36 to see how the schema is obtained. A test with a local Kafka topic and Avro schema would verify the fix.

Scritto dal modello di indicizzazione a partire dal testo della issue.