apache / apache/druid

ORC file sampler is not able to identify the date data type

Open
#16,573 3 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").

### Affected Version

We observed the issue from Druid version 26.0.0 and the same issue is there on the latest version which is 29.0.1
on Version 0.22.1 we are not seeing this issue.

### Description
we encountering an issue when trying to parse the ORC file created from a Hive external table on Druid version 27.0.0. The error message "Failed to sample data: Unhandled type: COMPLEX" is displayed. However, I can load the data into the cluster using the pre-created JSON spec, and the data is being received correctly. The error only occurs during the Parse data step. Interestingly, I can parse the same file on Druid version 0.22.1.
(The JSON spec used to load the data on version 27 was created from version 0.22.1)

we have also tried the latest version (29.0.1)of Druid, but we are still encountering the same error.

in this case it is an issue with the date type. our ORC reader returns it as a Joda DateTime object. our auto type detection code is not able to figure out what to do with this. the type detection code is expecting to see more primitive types like strings, numbers, & arrays. what changed in recent druid versions is that the sampler started using auto type detection.

![image](https://github.com/apache/druid/assets/45865769/e5e6983b-f913-44ef-a440-abf9c6ceedc0)
Screenshot 2024-06-07 at 4 22 04 PM

Contributor guide

Open the contributing guide

Research direction

Start with the ORC sampler's auto type detection used during the Parse data step and reproduce the failure with an ORC date column returned as a Joda DateTime object. Done means sampling recognizes the date value instead of reporting "Failed to sample data: Unhandled type: COMPLEX" while preserving the existing ability to load the data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.