Azure / Azure/Azure-DataFactory
Oracle to SQL pipeline copy table operation of type NUMBER in oracle results in precision loss
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
I have a pipeline set up to copy from ORACLE to SQL in Azure.
In ORACLE I have a table with a column that is set to type `NUMBER` containing values such as 1.2 or 2.345678
The maximum value will be xx.xxxxxxxx
I've set the copy operation to :
```
{
"source": {
"name": "MYDECIMALVALUE",
"type": "Decimal"
},
"sink": {
"name": "MyDecimalValue",
"type": "Decimal",
"physicalType": "decimal",
"precision": 12,
"scale": 8,
}
},
```
and even tried:
```
{
"source": {
"name": "MYDECIMALVALUE",
"type": "String"
},
"sink": {
"name": "MyDecimalValue",
"type": "String",
"physicalType": "varchar"
}
},
```
if relevant the section above:
```
"typeProperties": {
"enableStaging": false,
"source": {
"type": "OracleSource",
"partitionOption": "None",
"queryTimeout": "02:00:00"
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": null
},
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": false,
"treatBooleanAsNumber": false
},
```
However the resulting data in the SQL has precision errors, e.g. 1.2 might become 1.20000001 or 1.19999999
I have also found that some numbers come out like this:
`-0.00002` becomes `-2e-005` when imported to varchar, and then it can't be parsed into a number using CAST.
I'm wondering if this is a glitch or if there's some way to resolve it.
Further investigation has shown that the type NUMBER in oracle is I think floating point when there's no precision/scale set, although oracle seems to have no issue when doing a select *. Likewise within ADF, hitting the preview button shows the data as you'd expect to see it, it's only the migration itself that seems problematic.
I've put a hack in to this which is to change the copy table operation from Table to Query, and done `SELECT TO_CHAR(Column) as ColumnName`. With the sink still as a decimal, ADF automatically parses it to decimal and it works without precision issues, presumably because TO_CHAR is making oracle do the conversation instead of SQL.
So I think there's something going on under the hood that's a bug when the operation actually runs vs when you press the button to preview the data. However it could also be chalked up to, well, don't use a NUMBER in oracle if you want precise values. Unfortunately I don't have the ability to change the Oracle schema in any way given like many uses of ADF this is for moving a legacy database.
Contributor guide
Assessment
This issue has not been assessed yet.