Azure / Azure/Azure-DataFactory
Copy Activity Upsert fails when the sink dataset is a table with an IDENTITY column
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
An interim table is created during a Copy Activity upsert via a SELECT INTO statement, as described [here](https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database?tabs=data-factory#upsert-data). This is an example of how this is done, taken from the Trace file:
`IF OBJECT_ID('[##InterimTable_14d7d364-393c-4b28-bb92-7ee742827b85]', 'U') IS NULL BEGIN select * into [##InterimTable_14d7d364-393c-4b28-bb92-7ee742827b85] from [dbo].[foo_tbl] where 1 = 2 union select * from [dbo].[foo_tbl] where 1 = 2 alter table [##InterimTable_14d7d364-393c-4b28-bb92-7ee742827b85] add BatchIdentifier BIGINT IDENTITY(1,1) END`
The inclusion of a UNION in the SELECT INTO means that if the sink dataset has a surrogate key column with an IDENTITY property, this is not inherited by the interim table but the column is created with a NOT NULL constraint, as per [Microsoft documentation](https://docs.microsoft.com/en-us/sql/t-sql/queries/select-into-clause-transact-sql?redirectedfrom=MSDN&view=sql-server-ver15#data-types)
The NOT NULL constraint will cause an error since no source dataset column is mapped to the surrogate key column because it is expected to be populated via the IDENTITY property.
Contributor guide
Assessment
This issue has not been assessed yet.