Hive cannot determine the list of columns automatically based on Parquet serde
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Today we are not able to create a parquet based hive table without having to specify the column names and types. When we try to define it the following way, we get the error
"14/08/20 17:27:46 ERROR ql.Driver: FAILED: SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified"
CREATE TABLE parquet_test
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'/user/pratik/campaigns';
Whereas if we create a hive table on top of AVRO based files, we do not need to specify the column names, hive automatically figures out the schema through the SerDe.
CREATE EXTERNAL TABLE campaigns
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION '/user/pratik/campaigns'
TBLPROPERTIES ('avro.schema.url'='hdfs:///user/pratik/campaigns.avsc');
Reporter: Pratik Khadloya
Assignee: Ashish Singh / @SinghAsDev
Note: This issue was originally created as PARQUET-76. Please see the migration documentation for further details.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the original PARQUET-76 report and the migration documentation linked to PARQUET-2502 first. Confirm whether automatic column discovery for Parquet-backed Hive tables is still required or has moved elsewhere, then identify the current implementation and test entry points before defining the remaining scope and acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100