Azure / Azure/Azure-DataFactory

Oracle to SQL pipeline copy table operation of type NUMBER in oracle results in precision loss

Open
#527 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

I have a pipeline set up to copy from ORACLE to SQL in Azure.

In ORACLE I have a table with a column that is set to type `NUMBER` containing values such as 1.2 or 2.345678

The maximum value will be xx.xxxxxxxx

I've set the copy operation to :

```
{
"source": {
"name": "MYDECIMALVALUE",
"type": "Decimal"
},
"sink": {
"name": "MyDecimalValue",
"type": "Decimal",
"physicalType": "decimal",
"precision": 12,
"scale": 8,
}
},
```

and even tried:

```
{
"source": {
"name": "MYDECIMALVALUE",
"type": "String"
},
"sink": {
"name": "MyDecimalValue",
"type": "String",
"physicalType": "varchar"
}
},
```

if relevant the section above:

```
"typeProperties": {
"enableStaging": false,
"source": {
"type": "OracleSource",
"partitionOption": "None",
"queryTimeout": "02:00:00"
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": null
},
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": false,
"treatBooleanAsNumber": false
},
```

However the resulting data in the SQL has precision errors, e.g. 1.2 might become 1.20000001 or 1.19999999

I have also found that some numbers come out like this:

`-0.00002` becomes `-2e-005` when imported to varchar, and then it can't be parsed into a number using CAST.

I'm wondering if this is a glitch or if there's some way to resolve it.

Further investigation has shown that the type NUMBER in oracle is I think floating point when there's no precision/scale set, although oracle seems to have no issue when doing a select *. Likewise within ADF, hitting the preview button shows the data as you'd expect to see it, it's only the migration itself that seems problematic.

I've put a hack in to this which is to change the copy table operation from Table to Query, and done `SELECT TO_CHAR(Column) as ColumnName`. With the sink still as a decimal, ADF automatically parses it to decimal and it works without precision issues, presumably because TO_CHAR is making oracle do the conversation instead of SQL.

So I think there's something going on under the hood that's a bug when the operation actually runs vs when you press the button to preview the data. However it could also be chalked up to, well, don't use a NUMBER in oracle if you want precise values. Unfortunately I don't have the ability to change the Oracle schema in any way given like many uses of ADF this is for moving a legacy database.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.