Incorrect user-supplied Avro schema (--avroSchemaFilePath) causes dbeam to produce invalid avro files.
Open
Nobody has claimed this yet.
- #377 by @rulle-io — closed without merging
- Dominant language
- Java
- Stars
- 197
- Forks
- 56
- PR merge metrics
- No merged PRs in 30d
Description
When an Avro user-supplied Avro schema doesn't correspond to actual columns returned by SQL SELECT statement, dbeam produces avro data files, which cause exceptions be thrown when users try to read them.
Identified use-cases
1. Avro schema has less fields than SQL SELECT columns
* SQL SELECT has columns: COF_NAME, plus many others ...
* Avro schema has fields: COF_NAME (fewer than expected).
* This scenario produces an Avro file, which seems to OK,
* but an exception is thrown when one tries to read it.
* org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -50
2. Avro schema has other order of fields than SQL SELECT columns
* SQL SELECT has columns: COF_NAME, SIZE, TOTAL
* Avro schema has fields: TOTAL, COF_NAME, SIZE (another order).
* This scenario produces an Avro file, which seems to OK,
* but an exception is thrown when one tries to read it.
* java.lang.ArrayIndexOutOfBoundsException: Index -50 out of bounds for length 2
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the two mismatches described for --avroSchemaFilePath: fewer schema fields and a different field order than the SQL SELECT. Trace the SQL-to-Avro export path and review closed pull request #377 for related context. Done means dbeam no longer produces Avro files that readers reject for either mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100