apache / apache/parquet-java

Hive cannot determine the list of columns automatically based on Parquet serde

Open
#1,533 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Component: Parquet Priority: Critical Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Today we are not able to create a parquet based hive table without having to specify the column names and types. When we try to define it the following way, we get the error
"14/08/20 17:27:46 ERROR ql.Driver: FAILED: SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified"

CREATE  TABLE parquet_test
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
  '/user/pratik/campaigns';

Whereas if we create a hive table on top of AVRO based files, we do not need to specify the column names, hive automatically figures out the schema through the SerDe.

CREATE EXTERNAL TABLE campaigns
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION '/user/pratik/campaigns'
TBLPROPERTIES ('avro.schema.url'='hdfs:///user/pratik/campaigns.avsc');

Reporter: Pratik Khadloya
Assignee: Ashish Singh / @SinghAsDev

Note: This issue was originally created as PARQUET-76. Please see the migration documentation for further details.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the original PARQUET-76 report and the migration documentation linked to PARQUET-2502 first. Confirm whether automatic column discovery for Parquet-backed Hive tables is still required or has moved elsewhere, then identify the current implementation and test entry points before defining the remaining scope and acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.