apache / apache/arrow

StreamReader not supporting Date values

Open
#45,090 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I have been given a parquet file with the following schema:

```
Name: string
Ident: string
Date: date32[day]
EventTimestamp: timestamp[us]
```
I dumped out the schema using a simple python script:

```
#! /usr/bin/env python3
import pyarrow as pa
import pyarrow.parquet as pq
import sys

schema = pq.read_schema(sys.argv[1])
print(schema)
```
I was trying to figure out what C++ type is date32[day]? I've looked all over and it SEEMS like the underlying data type is int32_t. But if I try to read it using that type, I get:
```

terminate called after throwing an instance of 'parquet::ParquetException'
what(): Column converted type mismatch. Column 'Date' has converted type 'DATE' not 'INT_32'
```
I've also tried using a string but I get this error:

```
terminate called after throwing an instance of 'parquet::ParquetException'
what(): Column physical type mismatch. Column 'Date' has physical type 'INT32' not 'BYTE_ARRAY'
```
Here's my code, almost verbatim what is on the example page for Parquet. Nothing I try allows me to read the DATE type:
```

#include
#include "arrow/io/file.h"
#include "parquet/stream_reader.h"
#include "readfile.h"

int main(int argc, char **argv)
{

if (argc < 2)
{
std::cout << "Usage: " << argv[0] << " infile;

PARQUET_ASSIGN_OR_THROW(
infile,
arrow::io::ReadableFile::Open(input_filename));

parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)}
std::string name;
std::string ident;
int32_t date;
std::string ts_event_utc;

while (!os.eof())
{
os >> name >> ident >> date >> ts_event_utc >> parquet::EndRow;
}
```
Looking at the code in streamreader.cc, I see no reference to a DATE type in either the exceptions at the top of the file or the operators below.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.