Azure ML Command Job re-using code from previous Job
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
**Related command**
az ml job create --file train-job.yaml
**Describe the bug**
When submitting a command job with the Azure ML CLI extension, the code is being re-used when executing a command using "python -m". For example, if using a "training" script:
src/train.py
`print('hello world')`
and submit the command job with a yaml script including the command:
`command: |
python3 -m src.train`
the std_out.txt file will print 'hello world'. Re-running the same job but changing the "training" script to:
src/train.py
`print('foo bar')`
When the command job executes, the std_out.txt will still print 'hello world' rather than the expected 'foo bar'. Viewing the 'code' tab of the command job results shows that the "training" script has been updated, but the job will actually execute on the original version of the code.
**To Reproduce**
Execute a command job from the CLI with 'az ml job create' multiple times using the 'python -m' command while changing the training script with each execution.
**Expected behavior**
The command job executes the job with the current code
**Environment summary**
{
"azure-cli": "2.39.0",
"azure-cli-core": "2.39.0",
"azure-cli-telemetry": "1.0.6",
"extensions": {
"ml": "2.8.0"
}
}
Also tried with ml: 2.5.0
**Additional context**
The behavior seems like it is caching the code from the first execution of a job. Rather than using the current code that is passed in with the command, the ML Workspace seems to be using the original version of the code. Possibly similar to the ML v1 pipeline's 'allow re-use' command.
Contributor guide
Assessment
This issue has not been assessed yet.