DVLinkToSQL_IncrementalCopy - Not processed records and missing deletion records

Hello! We have implemented the incremental csv from synapse serverless to azure sql dedicated. We experience outdated records in target db, which do not appear in dynamics.

There are records that appear in the dedicated server which do not have a deletion record in synapse, even though there is no functional data in dynamics for it.

Shouldn't we have gotten a deletion record given this scenario? Even by searching this particular id we do not get the IsDelete = 'True' record.

There are also records that appear in the target db for which there is a deletion record in synapse, but it was not processed somehow.

We have implemented the pipeline as described in the documentation. Nonetheless, we want to validate if the Dedup_and_Merge step is being executed correctly based on the result logs we are getting (given the second scenario we are experiencing)

{ "resultSetCount": 0, "recordsAffected": 1062, "resultSets": [], "outputParameters": {}, "outputLogs": "--De-duplicate the data in dbo._new_purchline--\n-- Merge data from _new_purchline to purchline----\n", "outputLogsLocation": "", "outputTruncated": false, "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (UK South)", "executionDuration": 6, "durationInQueue": { "integrationRuntimeQueue": 10 }, "billingReference": { "activityType": "PipelineActivity", "billableDuration": [ { "meterType": "AzureIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }

Are the resultSetCount and resultSets attributes expected to have some values? As far as we can see, all the steps have 0 on both, despite all having run successfully.

Thanks!

Start by reviewing the documented incremental CSV pipeline and the Dedup_and_Merge step, using the supplied execution logs and screenshots as the reported cases. Compare records with and without IsDelete = 'True' between Synapse and the Azure SQL dedicated target, and determine whether resultSetCount and resultSets being zero is expected. Done means identifying the cause of the missing or unprocessed deletion records and explaining the relevant pipeline result fields.