redpanda-data / redpanda-data/connect
Add support for parquet logical types to parquet_encode processor
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
In some cases, users will need to specify the logical type in the schema field. Details here: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
For example, when using type: BYTE_ARRAY to encode a string value, they might want to set the logical type to STRING so decoders will be able to interpret it correctly. For example, given this config:
input:
generate:
mapping: root.test = "deadbeef"
count: 1
interval: 0s
pipeline:
processors:
- parquet_encode:
schema:
- name: test
type: BYTE_ARRAY
output:
file:
path: output.parquet
codec: all-bytes
will produce a parquet binary which, when decoded with parquet-tools will contain a base64-encoded value:
> docker run --rm -v$(pwd):/tmp/parquet nathanhowell/parquet-tools cat /tmp/parquet/output.parquet
test = ZGVhZGJlZWY=
however, if we change this line of code to n = parquet.String(), then parquet-tools will output test = deadbeef.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in internal/impl/parquet/processor_encode.go at the linked line and trace how each schema field is converted into a Parquet type. Compare the current BYTE_ARRAY behavior with the requested STRING logical type and the Parquet LogicalTypes documentation. Done means the schema can specify a logical type and the example output is decoded as deadbeef rather than base64.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100