Azure / Azure/azure-rest-api-specs
[Cost Management] [API Management] Azure Cost Management Package just throws first 1000 lines and not managing the paging from the cost management rest api
- Dominant language
- TypeSpec
- Stars
- 3.1k
- Forks
- 5.9k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 444
Description
Package Name: azure-mgmt-costmanagement
Package Version: 3.0.0
Operating System: Windows
Python Version: 3.9.6
Describe the bug
When you call the Usage method in python: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/costmanagement/azure-mgmt-costmanagement/azure/mgmt/costmanagement/operations/_query_operations.py
that is related to this REST api endpoint here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage
the python api is not looping to get next page using the nextLink attribute in the response described here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage#queryresult
so we get a 1000 rows maximum per execution and no way to manage the paging with this library
To Reproduce
from azure.mgmt.costmanagement import CostManagementClient
from azure.mgmt.costmanagement.models import QueryAggregation,QueryGrouping,QueryDataset,QueryDefinition,QueryTimePeriod,QueryFilter,QueryComparisonExpression
from azure.mgmt.resource import ResourceManagementClient
from azure.identity import DefaultAzureCredential
from IPython.display import display, HTML
from typing import ContextManager
import json
import pandas as pd
import datetime as dt
import calendar
import numpy as np
thedate = dt.datetime.combine(dt.date.today(), dt.time())
first = thedate.replace(day=1)
last = thedate.replace(day = calendar.monthrange(thedate.year, thedate.month)[1])
credential = DefaultAzureCredential()
subscription_id = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
scope = '/subscriptions/{}'.format(subscription_id)
client = ResourceManagementClient(credential, subscription_id)
cmgmtc = CostManagementClient(credential = credential)
"""
'ResourceGroup','ResourceGroupName','ResourceLocation',
'ConsumedService','ResourceType','ResourceId',
'MeterId','BillingMonth','MeterCategory',
'MeterSubcategory','Meter','AccountName',
'DepartmentName','SubscriptionId','SubscriptionName',
'ServiceName','ServiceTier','EnrollmentAccountName',
'BillingAccountId','ResourceGuid','BillingPeriod',
'InvoiceNumber','ChargeType','PublisherType',
'ReservationId','ReservationName','Frequency',
'PartNumber','CostAllocationRuleName','MarkupRuleName',
'PricingModel','BenefitId','BenefitName',''
"""
query_template = (
QueryDefinition(
type = "ActualCost"
, timeframe = "ThisMonth"
, dataset =
QueryDataset(
granularity = "Monthly"
, aggregation = {
"totalCost": QueryAggregation(name = "Cost", function = "Sum")
,"totalCostUSD": QueryAggregation(name = "CostUSD", function = "Sum")
}
, grouping = [
QueryGrouping(name = "ResourceGroupName", type = "Dimension")
,QueryGrouping(name = "ResourceId" , type = "Dimension")
,QueryGrouping(name = "ResourceType" , type = "Dimension")
]
, filter =
QueryFilter(
dimensions =
QueryComparisonExpression(
name = "ResourceGroupName"
, operator = "In"
, values = ["RESOURCE_GROUP"]
)
)
)
)
)
replaced_query = (
query_template.deserialize(
json.loads(
json.dumps(
query_template.serialize()
).replace('RESOURCE_GROUP','destination_rg')
)
)
)
result = cmgmtc.query.usage( scope = scope, parameters = replaced_query)
data = pd.DataFrame(result.rows, columns = list(map(lambda col: col.name, result.columns)))
data_sorted = data.sort_values(by='CostUSD' ,ascending = False)
data_filtered = data_sorted
pd.set_option('display.max_rows', data_filtered.shape[0]+1)
display(HTML(data_filtered.to_html()))
Expected behavior
As a python developer using this package I would expect the result to be an iterable so I can get all result pages not just the first one
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Contributor guide
Research direction
Start with sdk/costmanagement/azure-mgmt-costmanagement/azure/mgmt/costmanagement/operations/_query_operations.py and compare its Usage behavior with the Cost Management query/usage REST endpoint. Trace how the response’s nextLink is represented and how pagination could be exposed to callers. Done means a query can retrieve rows beyond the first 1,000-page response, with behavior checked against a response containing nextLink.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, openapi, python
- Domain
- api, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100