AvroUtils is converting incorrectly LogicalType Timestamps from long into Joda DateTimes
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Copied from the mailing list report:
I think the method AvroUtils.toBeamSchema has a not expected side effect.
I found out that, if you invoke it and then you run a pipeline of GenericRecords containing a timestamp (l tried with logical-type timestamp-millis), Beam converts such timestamp from long to org.joda.time.DateTime. Even if you don't apply any transformation to the pipeline.
Do you think it's a bug?
More details on how to reproduce here:
https://lists.apache.org/thread.html/r43fb2896e496b7493a962207eb3b95360abc30b9d091b26f110264d0%40%3Cuser.beam.apache.org%3E
Imported from Jira [BEAM-9863](https://issues.apache.org/jira/browse/BEAM-9863). Original Jira may contain additional context.
Reported by: iemejia.
Contributor guide
Research direction
Start with AvroUtils.toBeamSchema and reproduce the linked mailing-list report using GenericRecords with a timestamp-millis logical type. Compare the record before and after invoking the method, and confirm whether the timestamp remains a long rather than becoming a Joda DateTime without a pipeline transformation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100