apache / apache/arrow

[C++] Can't use StreamWriter with ToParquetSchema schema

Open
#29,105 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Hi there,

First of all, I'm not sure if I'm doing this correctly, as it took a bit of reverse engineering to figure this out. 

I'm using Arrow 4.0.1 on Ubuntu with C++.

I followed the streaming example and created:
```cpp

#include
#include
#include
#include
#include
#include
#include
#include

#include "arrow/io/file.h"
#include "parquet/exception.h"
#include "parquet/stream_reader.h"
#include "parquet/stream_writer.h"

std::shared_ptr GetSchema() {
parquet::schema::NodeVector fields;
fields.push_back(parquet::schema::PrimitiveNode::Make(
"int64_field", parquet::Repetition::OPTIONAL, parquet::Type::INT64,
parquet::ConvertedType::NONE));

return std::static_pointer_cast(
parquet::schema::GroupNode::Make("schema", parquet::Repetition::REQUIRED, fields));
}

int main() {
std::shared_ptr outfile;

PARQUET_ASSIGN_OR_THROW(
outfile,
arrow::io::FileOutputStream::Open("parquet-stream-api-example.parquet"));

parquet::WriterProperties::Builder builder;
parquet::StreamWriter os{parquet::ParquetFileWriter::Open(outfile, GetSchema(), builder.build())};

os << int64_t(10);

return 0;
}
```
The code terminates with:
```java

terminate called after throwing an instance of 'parquet::ParquetException'
what(): Column converted type mismatch. Column 'int64_field' has converted type[NONE] not 'INT_64'
```
What I'm not sure about is `parquet::ConvertedType::NONE` part. The example provides this value even for primitives, while it's my understanding that it's necessary? If I do provide it, the code works.

Now, to the reverse engineering part. I'm trying to write to Parquet using `StreamWriter`. `StreamWriter` requires `parquet::schema::{{GroupNode`}} as the schema, but I begin with `arrow::Schema` I [found](https://github.com/apache/arrow/blob/e990d177b1f1dec962315487682f613d46be573c/cpp/src/parquet/arrow/writer.cc#L442) that it can be converted to `{{parquet::SchemaDescriptor`}} using `parquet::arrow::ToParquetSchema `utility. Looking at the utility [implementation](https://github.com/apache/arrow/blob/85f192a45755b3f15653fdc0a8fbd788086e125f/cpp/src/parquet/arrow/schema.cc#L322) I can see that `logical_type` is set to `None` which equals to `parquet::ConvertedType::None` and hence the converted schema can't be used due to the issue I described above.
1. Do we need to provide `ConvertedType` even for primitives?
1. Is it a bug in the schema conversion utility or [ColumnCheck](https://github.com/apache/arrow/blob/8e43f23dcc6a9e630516228f110c48b64d13cec6/cpp/src/parquet/stream_writer.cc#L200) assert?
1. Or is it expected behavior, in this case, what's a suggested approach? Build Parquet schema instead of Arrow Schema?

Thank you,

Vasily.

**Reporter**: [Vasily Fomin](https://issues.apache.org/jira/browse/ARROW-13438)

**Note**: *This issue was originally created as [ARROW-13438](https://issues.apache.org/jira/browse/ARROW-13438). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the streaming example with the Arrow 4.0.1 C++ API and inspect parquet/stream_writer.cc at ColumnCheck. Compare that behavior with the schema conversion paths in cpp/src/parquet/arrow/schema.cc and writer.cc, then determine whether ToParquetSchema and StreamWriter agree on primitive converted types. Done means the expected schema contract and the required behavior are established for this example.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.