apache / apache/arrow

The metadata info will lost in parquet file schema after writing the parquet file by calling the FileSystemDataset::Write() method.

Open
#33,335 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

This issue can be reproduced by the following code.

auto format = std::make_shared();
auto fs = std::make_shared(fs::kNoTime);
FileSystemDatasetWriteOptions write_options;
write_options.file_write_options = format->DefaultWriteOptions();
write_options.filesystem = fs;
write_options.base_dir = "root";
write_options.partitioning = std::make_shared(schema({}));
write_options.basename_template = "\{i}.parquet";
auto metadata =
    std::shared_ptr(new KeyValueMetadata(\{"foo"}, \{"bar"}));
auto dataset_schema = schema(\{field("a", int64())}, metadata);
RecordBatchVector batches{
    ConstantArrayGenerator::Zeroes(kRowsPerBatch, dataset_schema)};
ASSERT_EQ(0, batches[0]~~>column(0)~~>null_count());
auto dataset = std::make_shared(dataset_schema, batches);
ASSERT_OK_AND_ASSIGN(auto scanner_builder, dataset->NewScan());
ASSERT_OK(scanner_builder->Project(
    \{compute::call("add", {compute::field_ref("a"), compute::literal(1)})},
    \{"a_plus_one"}));
ASSERT_OK_AND_ASSIGN(auto scanner, scanner_builder->Finish());

// Before write the schema has the metadata info.
ASSERT_EQ(1, dataset_schema->HasMetadata());
ASSERT_OK(FileSystemDataset::Write(write_options, scanner));

ASSERT_OK_AND_ASSIGN(auto dataset_factory, FileSystemDatasetFactory::Make(
                                               fs, \{"root/0.parquet"}, format, {}));
ASSERT_OK_AND_ASSIGN(auto written_dataset, dataset_factory->Finish(FinishOptions{}));

// After write the schema does not has the metadata info.
ASSERT_EQ(0, written_dataset->schema()->HasMetadata());

**Reporter**: [Ke Jia](https://issues.apache.org/jira/browse/ARROW-18140)

**Note**: *This issue was originally created as [ARROW-18140](https://issues.apache.org/jira/browse/ARROW-18140). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start at FileSystemDataset::Write and reproduce the behavior using the provided in-memory dataset, mock filesystem, and ParquetFileFormat setup. Compare the schema metadata before writing with written_dataset->schema() after reading; done when the metadata is preserved and both assertions pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.