Need an example of creating DDL for a Hive Parquet table with EEL
- Dominant language
- Scala
- Stars
- 147
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
- CSVSource to HiveSink
```scala
val schema = AvroSchemaFns.fromAvroSchema(new Schema.Parser().parse(new File("user.avsc")))
CsvSource(path)
.withSchema(schema)
.to(HiveSink("mydatabase", "myTable"))
```
- Table field: **fname**, **lname**, **age**, **salary**
- 2 partition keys of **country** and **city**
```scala
object EelCreateTableExample extends App {
val crateTableCommand = HiveDDL.showDDL(
tableName = "mydatabase.mytable",
partitions = Seq(
PartitionColumn("country", StringType),
PartitionColumn("city", StringType)
),
fields = Seq(
Field("fname", StringType),
Field("lname", StringType),
Field("age", IntType.Signed),
Field("salary", DecimalType(38, 5))
),
tableType = TableType.EXTERNAL_TABLE,
location = Some("hdfs://nameservice1/blah/mytable_location"),
serde = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
inputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
outputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
props = Map.empty,
tableComment = Some("my lovely table"),
ifNotExists = true
)
println(crateTableCommand)
}
```
- Ouput:
```
CREATE EXTERNAL TABLE IF NOT EXISTS `mydatabase.mytable` (
`fname` string,
`lname` string,
`age` int,
`salary` decimal(38,5))
PARTITIONED BY (
`country` string,
`city` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'hdfs://nameservice1/blah/mytable_location'
```
Contributor guide
No contributing guide indexed for this repository
Research direction
The example is already provided in the issue body. Look for the HiveDDL class in the codebase to understand how showDDL is implemented. Verify the output matches the expected CREATE EXTERNAL TABLE statement. Add this example to the project's documentation, likely in a examples or docs directory. Run the example to ensure it compiles and prints the correct DDL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala
- Domain
- data-engineering, databases
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100