"az consumption usage list" command cannot be run in parallel
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
**Related command**
az consumption usage list
**Describe the bug**
The command can't be parallelized. You essentially have to run it serially (well, sometimes I can do 2, maybe 3 at once... woohoo!) Combined with the very long execution time (https://github.com/Azure/azure-cli/issues/23204), this means I can't run daily reports. All other az-cli commands support at least a tiny bit of parallelization, most easily handling fifty or more jobs at a time.
Running this sub command in parallel (20 at a time) over our 1059 subscriptions returned:
63 successful
23 Returned no output (perhaps succeeded, perhaps not, haven't investigated)
973 failures
Failure breakdown according to the ERROR messages given by az-cli -
926 ERROR: (429) Too many requests. Please retry after 60 seconds.
44 ERROR: (SubscriptionNotFound)
2 ERROR: (401) Unauthorized
1 ERROR: (400) Subscription scope usage is not supported for current api version.
The 44 that weren't found... a bit odd, since this was run immediately after running `az account list`
to get the subscriptions in the tenant.
On the bright side the failed run only took a bit over 4 minutes to complete.
**To Reproduce**
Here I'm using the `rush` command to parallelize shell commands, as I do with all the other az-cli sub commands I use (rush is at https://github.com/shenwei356/rush); you can run from 1-N jobs in parallel at a time; when it finishes one job it starts another. Here rush is reading from standard input... the file "subs.txt", which has, one per line, the IDs of the > 1000 subscriptions we run in this tenant.)
First, the simple "costs.sh" shell script that will be run in parallel; it puts the result in /tmp -
```
start=$(gdate -d "last month" "+%Y-%m-01")
end=$(gdate -d "$(date +%Y/%m/01) - 1 day 00:00" +"%Y-%m-%d")
date_filter="-s $start -e $end"
az consumption usage list -a -m $date_filter --subscription "$1" > /tmp/$1.res
```
The output of running the above script under `rush` is attached (gitout.txt) the -j option below specifies the number of jobs to run in parallel (20 here.)
(see attachment)
[gitout.txt](https://github.com/Azure/azure-cli/files/9136010/gitout.txt)
**Expected behavior**
The command should be able to be run in parallel from a single endpoint and return the results within a reasonable time frame. The cost web UI, which has the costs sliced and diced, actually loads pretty quickly.
**Environment summary**
Installed via `brew` on a macbook and iMac, currently running MacOS 10.16 and/or 10.15.7, but this has been slow for literally the last few years no matter what the OS or az-cli versions. Here's the macbook version, which was used for the above test and output -
`az version
{
"azure-cli": "2.23.0",
"azure-cli-core": "2.23.0",
"azure-cli-telemetry": "1.0.6",
"extensions": {
"azure-devops": "0.18.0",
"databricks": "0.7.2",
"kusto": "0.2.0",
"log-analytics-solution": "0.1.1",
"resource-graph": "1.1.0",
"storage-preview": "0.7.2"
}
}
```
**Additional context**
The `consumption` sub command has been in preview for years. Come on. From the consumptio:
"WARNING: Command group 'consumption' is in preview and under development."
I can send a full debug log of a run (about 500-800MB) if you want (I can point to a URI or whatever, if you mail me.)
Contributor guide
Assessment
This issue has not been assessed yet.