microsoft / microsoft/finops-toolkit

config_runbackfilljob fails to run using datafactory studio to run the job

Open
#1,072 4 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Skill: Data factory Skill: Documentation Tool: FinOps hubs
Dominant language
PowerShell
Stars
603
Forks
249
Avg merge
7d 11h
Merged PRs (30d)
11

Description

🐛 Problem

running pipeline config_runbackfilljob in data factory studio fails to run. Set the scope config in settings.json however once you try run the job you get a myriad of errors

One was export time cannot exceed 84 months
One was startdate and enddate must be in the same month and another was
Either both startDate and endDate needs to be present or both should not be present."

Date format used for the above tests was DD/MM/YYYY

On Second round of testing using a single month with date format MM/DD/YYYY it worked for one month only

Using the same MM/DD/YYYY and spanning multiple months it fails, this was a backfill pipeline and has a variable start and end date, you would assume you could put in start date 13 months ago and end date current date to get 13 months of backfill but that is not working.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing config_runbackfilljob from Azure Data Factory Studio using the settings.json scope configuration and the reported DD/MM/YYYY and MM/DD/YYYY date ranges. Compare single-month and multi-month backfills, then define done as allowing a valid multi-month startDate and endDate without the conflicting validation errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, powershell
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.