opensearch-project / opensearch-project/data-prepper
[BUG] DynamoDB source export converts Numbers ending in 0 to scientific notation
@san81 is already working on this.
Since May 21, 2024.
- Dominant language
- Java
- Stars
- 374
- Forks
- 355
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
Given an item in with a Number type ending in 0, such as 1702062202420, the DynamoDB source will convert it to scientific notation for export items.
{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 702062202420 }
To Reproduce
Steps to reproduce the behavior:
- Create a pipeline with a dynamodb source with export and an opensearch
- Once the export is complete, the document will be sent to the OpenSearch sink as the following with the
my_number_ending_in_0key converted to scientific notation
{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 1.70206220242E+12 }
Expected behavior
The Numbers ending in 0 should not be manipulated and the above example should result in
{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 702062202420 }
Additional context
The conversion only happens for export values when converting from the ion line here (https://github.com/opensearch-project/data-prepper/blob/91ff22d6da2b14d8a27ade89ee516341181c8bd6/data-prepper-plugins/dynamodb-source/src/main/java/org/opensearch/dataprepper/plugins/source/dynamodb/converter/ExportRecordConverter.java#L82), but Data Prepper JacksonEvent also converts to scientific notation when converting to json string (https://github.com/opensearch-project/data-prepper/blob/91ff22d6da2b14d8a27ade89ee516341181c8bd6/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java#L621). I initially had created a custom deserializer that iterated over and converted all decimals of this format to not use scientific notation, however this may not be the best approach
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.