Azure / Azure/azure-rest-api-specs

Azure Cost Management Package just throws first 1000 lines and not managing the paging from the cost management rest api

Open
#22,528 7 comments 0 reactions 0 assignees View on GitHub
Consumption - Billing customer-reported question Service Attention
Dominant language
TypeSpec
Stars
3.1k
Forks
5.9k
Avg merge
2d 22h
Merged PRs (30d)
444

Description

Package Name: azure-mgmt-costmanagement
Package Version: 3.0.0
Operating System: Windows
Python Version: 3.9.6
Describe the bug
When you call the Usage method in python: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/costmanagement/azure-mgmt-costmanagement/azure/mgmt/costmanagement/operations/_query_operations.py
that is related to this REST api endpoint here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage
the python api is not looping to get next page using the nextLink attribute in the response described here https://docs.microsoft.com/es-es/rest/api/cost-management/query/usage#queryresult
so we get a 1000 rows maximum per execution and no way to manage the paging with this library

To Reproduce

from azure.mgmt.costmanagement import CostManagementClient
from azure.mgmt.costmanagement.models import QueryAggregation,QueryGrouping,QueryDataset,QueryDefinition,QueryTimePeriod,QueryFilter,QueryComparisonExpression
from azure.mgmt.resource import ResourceManagementClient
from azure.identity import DefaultAzureCredential
from IPython.display import display, HTML
from typing import ContextManager

import json
import pandas as pd
import datetime as dt
import calendar
import numpy as np

thedate = dt.datetime.combine(dt.date.today(), dt.time())
first = thedate.replace(day=1)
last = thedate.replace(day = calendar.monthrange(thedate.year, thedate.month)[1])

credential = DefaultAzureCredential()
subscription_id = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

scope = '/subscriptions/{}'.format(subscription_id)

client = ResourceManagementClient(credential, subscription_id)

cmgmtc = CostManagementClient(credential = credential)

"""
'ResourceGroup','ResourceGroupName','ResourceLocation',
'ConsumedService','ResourceType','ResourceId',
'MeterId','BillingMonth','MeterCategory',
'MeterSubcategory','Meter','AccountName',
'DepartmentName','SubscriptionId','SubscriptionName',
'ServiceName','ServiceTier','EnrollmentAccountName',
'BillingAccountId','ResourceGuid','BillingPeriod',
'InvoiceNumber','ChargeType','PublisherType',
'ReservationId','ReservationName','Frequency',
'PartNumber','CostAllocationRuleName','MarkupRuleName',
'PricingModel','BenefitId','BenefitName',''
"""

query_template = (
QueryDefinition(
type = "ActualCost"
, timeframe = "ThisMonth"
, dataset =
QueryDataset(
granularity = "Monthly"
, aggregation = {
"totalCost": QueryAggregation(name = "Cost", function = "Sum")
,"totalCostUSD": QueryAggregation(name = "CostUSD", function = "Sum")
}
, grouping = [
QueryGrouping(name = "ResourceGroupName", type = "Dimension")
,QueryGrouping(name = "ResourceId" , type = "Dimension")
,QueryGrouping(name = "ResourceType" , type = "Dimension")
]
, filter =
QueryFilter(
dimensions =
QueryComparisonExpression(
name = "ResourceGroupName"
, operator = "In"
, values = ["RESOURCE_GROUP"]
)
)
)
)
)

replaced_query = (
query_template.deserialize(
json.loads(
json.dumps(
query_template.serialize()
).replace('RESOURCE_GROUP','destination_rg')
)
)
)

result = cmgmtc.query.usage( scope = scope, parameters = replaced_query)

data = pd.DataFrame(result.rows, columns = list(map(lambda col: col.name, result.columns)))

data_sorted = data.sort_values(by='CostUSD' ,ascending = False)

data_filtered = data_sorted

pd.set_option('display.max_rows', data_filtered.shape[0]+1)

display(HTML(data_filtered.to_html()))

Expected behavior
As a python developer using this package I would expect the result to be an iterable so I can get all result pages not just the first one

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

- This has been reported here https://github.com/Azure/azure-sdk-for-python/issues/23175 as well. The previous entries do not mention if this will be fixed or added as an enhancement request.

Contributor guide

Open the contributing guide

Research direction

Start with azure/mgmt/costmanagement/operations/_query_operations.py and the Cost Management QueryResult documentation linked in the issue, then compare the related azure-sdk-for-python issue #23175. Confirm how the nextLink attribute is exposed and identify the package entry point that should provide subsequent pages; done means callers can retrieve all result rows rather than only the first 1000.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.