cockroachdb / cockroachdb/cockroach
parquet: row groups only contain single data page
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
It appears that after https://github.com/cockroachdb/cockroach/pull/111138, we would only create a single data page per row group:
* https://github.com/cockroachdb/cockroach/blob/42025fb99b792ed55d243c309d71aaab93620741/pkg/util/parquet/writer.go#L165-L169
* https://github.com/cockroachdb/cockroach/blob/42025fb99b792ed55d243c309d71aaab93620741/pkg/util/parquet/writer.go#L195
For export, there is currently a single row group per file with a single data page:
* https://github.com/cockroachdb/cockroach/blob/d41451834131032b91dd0dd1463c692efc14e3d5/pkg/sql/importer/exportparquet.go#L148-L153
* https://github.com/cockroachdb/cockroach/blob/d41451834131032b91dd0dd1463c692efc14e3d5/pkg/sql/importer/exportparquet.go#L191-L194
* https://github.com/cockroachdb/cockroach/blob/049c30ac6956fa8f307ba2adae8b4388f5907859/pkg/util/parquet/writer.go#L280-L288
For changefeeds, there are multiple row groups per file but they also only have a single data page:
* https://github.com/cockroachdb/cockroach/blob/774f2a133d6002a96ade499c528a2d414a20505b/pkg/ccl/changefeedccl/parquet_sink_cloudstorage.go#L217-L224
* https://github.com/cockroachdb/cockroach/blob/85cfb844b07da0bb777cd050b4fe295ef4873428/pkg/ccl/changefeedccl/parquet.go#L161-L163
* https://github.com/cockroachdb/cockroach/blob/049c30ac6956fa8f307ba2adae8b4388f5907859/pkg/util/parquet/writer.go#L268-L276
This seems like undesirable behavior given that data pages should be small to allow for easier lookups. We should either use the default (i.e. 1 MiB) or let the user configure the data page size. Note the parquet docs (https://parquet.apache.org/docs/file-format/configurations) recommend 512MB - 1GB row groups and 8KB page sizes.
Related to: #140030
Jira issue: CRDB-47151
Contributor guide
Assessment
This issue has not been assessed yet.