Support for read and write of parquet file with lz4_raw compression codec in velox
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
From spark 3.5 parquet has been upgraded to 1.13.0 which supports LZ4_RAW codec. Spark now supports read and write of parquet data with this codec https://github.com/apache/spark/pull/41507. Change in velox is needed to support read and write of file with this codec in native.
While reading it maps it to LZ4
case thrift::CompressionCodec::LZ4_RAW:
return common::CompressionKind::CompressionKind_LZ4;
and fails with this stack trace
01:04:52.170 ERROR org.apache.spark.util.TaskResources: Task 94 failed by error:
org.apache.gluten.exception.GlutenException: java.lang.RuntimeException: Exception: VeloxException
Error Source: RUNTIME
Error Code: UNKNOWN
Reason: [Range Constraint Violation : 80>=4030799104] : {} decompression failed, remainingOutputSize is less than decompressedBlockSize, remainingOutputSize: {}, decompressedBlockSize: {}5804030799104
Retriable: False
Expression: remainingOutputSize >= decompressedBlockSize
Context: Split [Hive: file:///tmp/spark-dd5b62c5-00b8-4f3d-815c-337ac74e1916/part-00002-a44b5206-3087-4464-902c-a11df39ae28f-c000.lz4raw.parquet 0 - 846] Task Gluten_Stage_47_TID_94
Top-Level Context: Same as context.
Function: decompress
File: /home/workspace/gluten/ep/build-velox/build/velox_ep/velox/dwio/common/compression/Compression.cpp
Line: 258
Stack trace:
0 facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, bool, facebook::velox::VeloxException::Type, std::basic_string_view >)
1 facebook::velox::dwio::common::exception::LoggedException::LoggedException(char const*, unsigned long, char const*, char const*, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&)
2 facebook::velox::dwio::common::compression::(anonymous namespace)::LzoAndLz4DecompressorCommon::decompress(char const*, unsigned long, char*, unsigned long) [clone .cold]
3 facebook::velox::dwio::common::compression::PagedInputStream::readOrSkip(void const**, int*)
4 facebook::velox::dwio::common::SeekableInputStream::readFully(char*, unsigned long)
5 facebook::velox::parquet::PageReader::decompressData(char const*, unsigned int, unsigned int)
Contributor guide
Research direction
Start at velox/dwio/common/compression/Compression.cpp:258 and the LZ4_RAW mapping shown in the issue, then trace parquet PageReader::decompressData. Verify native read and write paths with Spark 3.5 parquet files using LZ4_RAW; done when both operations work without the reported decompression failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100