51zero / 51zero/eel-sdk

Support Hive MAP columns

Aperta
#388 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Scala
Stelle
147
Fork
32
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

I get this error when using HiveSink to write to a Hive table with a `MAP` column.

```
java.lang.RuntimeException: Unsupported hive type [map]
at scala.sys.package$.error(package.scala:27)
at io.eels.component.hive.HiveSchemaFns$.fromHiveType(HiveSchemaFns.scala:56)
at io.eels.component.hive.HiveSchemaFns$.fromHive(HiveSchemaFns.scala:25)
at io.eels.component.hive.HiveSchemaFns$.fromHiveField(HiveSchemaFns.scala:22)
at io.eels.component.hive.HiveOps$$anonfun$3.apply(HiveOps.scala:144)
at io.eels.component.hive.HiveOps$$anonfun$3.apply(HiveOps.scala:144)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at io.eels.component.hive.HiveOps.schema(HiveOps.scala:144)
at io.eels.component.hive.HiveSink.open(HiveSink.scala:124)
at io.eels.component.hive.HiveSink$$anonfun$open$1.apply(HiveSink.scala:88)
at io.eels.component.hive.HiveSink$$anonfun$open$1.apply(HiveSink.scala:88)
at scala.collection.generic.GenTraversableFactory.tabulate(GenTraversableFactory.scala:148)
at io.eels.component.hive.HiveSink.open(HiveSink.scala:88)
at io.eels.datastream.SinkAction.execute(SinkAction.scala:46)
at io.eels.datastream.DataStream$class.to(DataStream.scala:727)
at io.eels.datastream.DataStream$$anon$50.to(DataStream.scala:772)
at io.eels.datastream.DataStream$class.to(DataStream.scala:726)
at io.eels.datastream.DataStream$$anon$50.to(DataStream.scala:772)
...
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

The error originates in HiveSchemaFns.scala at line 56, where the fromHiveType function does not handle the MAP type. Examine the Hive type system mapping in that file and the surrounding schema conversion logic. Look for existing support for other complex types like ARRAY or STRUCT to understand the pattern. Adding MAP support will likely involve updating the type matching and creating an appropriate EEL schema field.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
scala
Ambito
data-engineering, databases
Tipo di issue
Funzionalità
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Specificata chiaramente
Idoneità per principianti
40/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.