Handling Decimal / Numeric Data - Singer.Decimal Standards

There has been some discussion around how to handle Floats, Decimal and Numeric data.

We were finding that decimal data was ending up as floats in target-snowflake from various database taps. The taps were opinionated using the MultipleOf etc, but the results were in the wrong data type and we were finding there was truncation / rounding. Part of the issue was to do with the Python Language not supporting large numeric data leading to truncation.

We have learned about the singer.decimal format which doesn't take an opinion on the output format, leaving the target to determine how to treat the string representation of the numeric data. The problem with this is you loose a lot of automation, like the automated creation of tables or addition of columns with the correct datatype.

The problem is there needs to be further information provided to allow the correct creation of datatypes by the target as part of the schema message i.e. information about the precision and scale if it is available at source.

We have elected to add some AdditionalProperties - providing the Scale and Precision to help with decimal data. See the slack message below.

https://app.slack.com/client/TFG99TU9K/search/search-eyJkIjoiZGVjaW1hbCUyMGRhdGEiLCJxIjoiVTAyQjMxUUJTVUIiLCJyIjoiZGVjaW1hbCUyMGRhdGEifQ==/thread/C013EKWA2Q1-1650990076.383639

While we have taken an opinionate view on how to implement this, it would be great if there was some standards around how to navigate the challenges of emitting numeric data without data loss. We would also be interested in views around our approach to dealing with this issue.

Start by reviewing the issue's discussion of Singer.Decimal, Python numeric limitations, and the linked Slack thread. Compare the proposed precision and scale AdditionalProperties with the requirements for target table creation and lossless numeric handling. Done means the working group agrees on a documented standard or a clearly scoped next step.