Azure / Azure/Azure-DataFactory
ADF doesn't recognise changes pushed via git for about 10 minutes
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
I have ADF with a pipeline that copies data from an oracle to a sql database.
All of my config (json files for ADF) are generated locally.
Every time I make a change and push it, ADF doesn't update properly. I can see my changes within ADF clicking on pipelines etc but when I tell it to run the pipeline, I get errors that shouldn't exist.
For example I recently renamed a column in my target SQL database from KittensId to KittenId and pushed it. Within my pipeline json I can see it is KittenId, the same within my dataset definition. But when I run the pipeline, I get the error KittensId is not a column in the target.
This problem has happened repeatedly throughout development. It seems to be caching.
I launch my pipeline by clicking Add Trigger > Trigger Now.
After running it over and over it seems after about 10 minutes that it updates.
Can anyone advise how I can fix this issue? Can I get it to recognise I've pushed an updated config quicker?
Contributor guide
Assessment
This issue has not been assessed yet.