Azure / Azure/Azure-DataFactory

ADF doesn't recognise changes pushed via git for about 10 minutes

Open
#420 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

I have ADF with a pipeline that copies data from an oracle to a sql database.

All of my config (json files for ADF) are generated locally.

Every time I make a change and push it, ADF doesn't update properly. I can see my changes within ADF clicking on pipelines etc but when I tell it to run the pipeline, I get errors that shouldn't exist.

For example I recently renamed a column in my target SQL database from KittensId to KittenId and pushed it. Within my pipeline json I can see it is KittenId, the same within my dataset definition. But when I run the pipeline, I get the error KittensId is not a column in the target.

This problem has happened repeatedly throughout development. It seems to be caching.

I launch my pipeline by clicking Add Trigger > Trigger Now.

After running it over and over it seems after about 10 minutes that it updates.

Can anyone advise how I can fix this issue? Can I get it to recognise I've pushed an updated config quicker?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.