apache / apache/amoro

[Improvement]: When manually setting table.event-time-field to a field, it should widen its data file mertic length

Open
#1,604 3 comments 0 reactions 0 assignees View on GitHub
good first issue not-stale type:improvement
Dominant language
Java
Stars
1.2k
Forks
395
Avg merge
4d 10h
Merged PRs (30d)
33

Description

### Search before asking

- [X] I have searched in the [issues](https://github.com/NetEase/arctic/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

The current default length of the iceberg table for metadata metric is 'write.metadata.metrics.default' = 'truncate(16)'. meaning that data larger than 16 will have a loss of precision.
When we set table.event-time-field we often choose a time field in the format yyyy-MM-dd HH:mm:ss, obviously this string is longer than 16, then there will be a loss of precision when parsing the upperbounds and lowerbounds of the datafile.
From the user's point of view, most of the cases will not be able to change 'write.metadata.metrics.default' to a larger value or to 'full', so it is necessary to help the user to set this configuration.

### How should we improve?

AMS should set global default configuration such as 'write.metadata.metrics.default' to 'truncate(32)' when creating a table, or just for the fields set in the table.event-time-field configuration mertric changes

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Subtasks

_No response_

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing AMS table creation and handling of the table.event-time-field configuration, then inspect how the Iceberg metadata metric default is applied. Compare the global and field-specific approaches described in the issue and add focused coverage for event-time fields longer than 16 characters. Done means upper and lower bounds retain the required precision without users changing the default configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.