Azure / Azure/Azure-DataFactory

Copy Activity Upsert fails when the sink dataset is a table with an IDENTITY column

Open
#388 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

An interim table is created during a Copy Activity upsert via a SELECT INTO statement, as described [here](https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database?tabs=data-factory#upsert-data). This is an example of how this is done, taken from the Trace file:

`IF OBJECT_ID('[##InterimTable_14d7d364-393c-4b28-bb92-7ee742827b85]', 'U') IS NULL BEGIN select * into [##InterimTable_14d7d364-393c-4b28-bb92-7ee742827b85] from [dbo].[foo_tbl] where 1 = 2 union select * from [dbo].[foo_tbl] where 1 = 2 alter table [##InterimTable_14d7d364-393c-4b28-bb92-7ee742827b85] add BatchIdentifier BIGINT IDENTITY(1,1) END`

The inclusion of a UNION in the SELECT INTO means that if the sink dataset has a surrogate key column with an IDENTITY property, this is not inherited by the interim table but the column is created with a NOT NULL constraint, as per [Microsoft documentation](https://docs.microsoft.com/en-us/sql/t-sql/queries/select-into-clause-transact-sql?redirectedfrom=MSDN&view=sql-server-ver15#data-types)

The NOT NULL constraint will cause an error since no source dataset column is mapped to the surrogate key column because it is expected to be populated via the IDENTITY property.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.