apache / apache/beam

AvroUtils is converting incorrectly LogicalType Timestamps from long into Joda DateTimes

Open
#20,205 2 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Copied from the mailing list report:

I think the method AvroUtils.toBeamSchema has a not expected side effect.
I found out that, if you invoke it and then you run a pipeline of GenericRecords containing a timestamp (l tried with logical-type timestamp-millis), Beam converts such timestamp from long to org.joda.time.DateTime. Even if you don't apply any transformation to the pipeline.
Do you think it's a bug?

More details on how to reproduce here:

https://lists.apache.org/thread.html/r43fb2896e496b7493a962207eb3b95360abc30b9d091b26f110264d0%40%3Cuser.beam.apache.org%3E

Imported from Jira [BEAM-9863](https://issues.apache.org/jira/browse/BEAM-9863). Original Jira may contain additional context.
Reported by: iemejia.

Contributor guide

Open the contributing guide

Research direction

Start with AvroUtils.toBeamSchema and reproduce the linked mailing-list report using GenericRecords with a timestamp-millis logical type. Compare the record before and after invoking the method, and confirm whether the timestamp remains a long rather than becoming a Joda DateTime without a pipeline transformation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.