[C++] Can't use StreamWriter with ToParquetSchema schema
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Hi there,
First of all, I'm not sure if I'm doing this correctly, as it took a bit of reverse engineering to figure this out.
I'm using Arrow 4.0.1 on Ubuntu with C++.
I followed the streaming example and created:
```cpp
#include
#include
#include
#include
#include
#include
#include
#include
#include "arrow/io/file.h"
#include "parquet/exception.h"
#include "parquet/stream_reader.h"
#include "parquet/stream_writer.h"
std::shared_ptr GetSchema() {
parquet::schema::NodeVector fields;
fields.push_back(parquet::schema::PrimitiveNode::Make(
"int64_field", parquet::Repetition::OPTIONAL, parquet::Type::INT64,
parquet::ConvertedType::NONE));
return std::static_pointer_cast(
parquet::schema::GroupNode::Make("schema", parquet::Repetition::REQUIRED, fields));
}
int main() {
std::shared_ptr outfile;
PARQUET_ASSIGN_OR_THROW(
outfile,
arrow::io::FileOutputStream::Open("parquet-stream-api-example.parquet"));
parquet::WriterProperties::Builder builder;
parquet::StreamWriter os{parquet::ParquetFileWriter::Open(outfile, GetSchema(), builder.build())};
os << int64_t(10);
return 0;
}
```
The code terminates with:
```java
terminate called after throwing an instance of 'parquet::ParquetException'
what(): Column converted type mismatch. Column 'int64_field' has converted type[NONE] not 'INT_64'
```
What I'm not sure about is `parquet::ConvertedType::NONE` part. The example provides this value even for primitives, while it's my understanding that it's necessary? If I do provide it, the code works.
Now, to the reverse engineering part. I'm trying to write to Parquet using `StreamWriter`. `StreamWriter` requires `parquet::schema::{{GroupNode`}} as the schema, but I begin with `arrow::Schema` I [found](https://github.com/apache/arrow/blob/e990d177b1f1dec962315487682f613d46be573c/cpp/src/parquet/arrow/writer.cc#L442) that it can be converted to `{{parquet::SchemaDescriptor`}} using `parquet::arrow::ToParquetSchema `utility. Looking at the utility [implementation](https://github.com/apache/arrow/blob/85f192a45755b3f15653fdc0a8fbd788086e125f/cpp/src/parquet/arrow/schema.cc#L322) I can see that `logical_type` is set to `None` which equals to `parquet::ConvertedType::None` and hence the converted schema can't be used due to the issue I described above.
1. Do we need to provide `ConvertedType` even for primitives?
1. Is it a bug in the schema conversion utility or [ColumnCheck](https://github.com/apache/arrow/blob/8e43f23dcc6a9e630516228f110c48b64d13cec6/cpp/src/parquet/stream_writer.cc#L200) assert?
1. Or is it expected behavior, in this case, what's a suggested approach? Build Parquet schema instead of Arrow Schema?
Thank you,
Vasily.
**Reporter**: [Vasily Fomin](https://issues.apache.org/jira/browse/ARROW-13438)
**Note**: *This issue was originally created as [ARROW-13438](https://issues.apache.org/jira/browse/ARROW-13438). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the streaming example with the Arrow 4.0.1 C++ API and inspect parquet/stream_writer.cc at ColumnCheck. Compare that behavior with the schema conversion paths in cpp/src/parquet/arrow/schema.cc and writer.cc, then determine whether ToParquetSchema and StreamWriter agree on primitive converted types. Done means the expected schema contract and the required behavior are established for this example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100