opensearch-project / opensearch-project/data-prepper

[BUG] DynamoDB source export converts Numbers ending in 0 to scientific notation

Open
#3,840 6 comments 0 reactions 1 assignee View on GitHub

@san81 is already working on this.

Since May 21, 2024.

bug Priority-High
Dominant language
Java
Stars
374
Forks
355
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
Given an item in with a Number type ending in 0, such as 1702062202420, the DynamoDB source will convert it to scientific notation for export items.

{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 702062202420 }

To Reproduce
Steps to reproduce the behavior:

  1. Create a pipeline with a dynamodb source with export and an opensearch
  2. Once the export is complete, the document will be sent to the OpenSearch sink as the following with the my_number_ending_in_0 key converted to scientific notation
{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 1.70206220242E+12 }

Expected behavior
The Numbers ending in 0 should not be manipulated and the above example should result in

{"pk": "my_partition_key", "sk":"my_sort_key", "my_number_ending_in_0": 702062202420 }

Additional context
The conversion only happens for export values when converting from the ion line here (https://github.com/opensearch-project/data-prepper/blob/91ff22d6da2b14d8a27ade89ee516341181c8bd6/data-prepper-plugins/dynamodb-source/src/main/java/org/opensearch/dataprepper/plugins/source/dynamodb/converter/ExportRecordConverter.java#L82), but Data Prepper JacksonEvent also converts to scientific notation when converting to json string (https://github.com/opensearch-project/data-prepper/blob/91ff22d6da2b14d8a27ade89ee516341181c8bd6/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/JacksonEvent.java#L621). I initially had created a custom deserializer that iterated over and converted all decimals of this format to not use scientific notation, however this may not be the best approach

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.