microsoft / microsoft/azure-devops-python-api

Finding a specific build log is incredibly slow

未關閉
#486 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

主要語言
Python
星號
684
分支
218
平均合併
8 天 10 小時
30 天內合併 PR
1

描述

I have a script where I am trying to retrieve build logs for all PRs we have that are failing so I can parse the logs and collate the data.

I can get all of the PRs that are failing a particular check with the Github API pretty easily. I end up with detail links which all point to our internal ADO project.

It when I need to pull the logs from ADO that things get really slow. The code basically looks like this:

# PHASE 2: Collect logs for failed LintDiff runs from Azure DevOps
failure_logs = []
ado = get_ado_client()
for run in failed_lint_diff_runs:
    link = run.details_url
    build_id = get_build_id(link)
    build_client = ado.clients_v7_1.get_build_client()
    build_logs = build_client.get_build_logs("internal", build_id)
    for log in build_logs:
        lines = build_client.get_build_log_lines("internal", build_id, log.id)
        if "Starting: LintDiff" in lines[0]:
            print(f"Found LintDiff log for build {build_id} at log {log.id}")
            failure_logs.append(lines)
            break

When I run this, it takes about 20 minutes to run... all of the time is spent repeatedly calling get_build_log_lines to examine just the first line of the build log to see if it is the one I care about. The output looks like:

PRs with LintDiff failures: 16
Found LintDiff log for build 3497966 at log 297
Found LintDiff log for build 3493487 at log 375
Found LintDiff log for build 3497244 at log 242
Found LintDiff log for build 3471732 at log 295
Found LintDiff log for build 3468386 at log 417
Found LintDiff log for build 3468168 at log 395
Found LintDiff log for build 3459808 at log 307
Found LintDiff log for build 3475219 at log 296
Found LintDiff log for build 3469709 at log 307
Found LintDiff log for build 3497648 at log 310
Found LintDiff log for build 3462035 at log 212
Found LintDiff log for build 3370030 at log 342
Found LintDiff log for build 3349947 at log 391
Found LintDiff log for build 3328138 at log 306
Found LintDiff log for build 3327852 at log 304
Found LintDiff log for build 3327331 at log 306

You can see, we have a lot of build logs, so much of the time is wasted. I couldn't find anything about the build log object that is identifiable. The ID changes all the time, and while in the UI it has a consistent name, that name doesn't seem to be included here. If it were, it would save a ton of time.

Is there a better way to collect this information?

貢獻指南

這個儲存庫沒有索引到貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

從 issue 中展示的 Azure DevOps Python 用戶端呼叫開始:get_build_client()、get_build_logs() 和 get_build_log_lines()。檢查相應的 build-log API 行為,以及可用於在下載記錄行之前識別記錄的中繼資料。完成的標準是:無需反覆擷取每個記錄的內容,即可找到目標 LintDiff 記錄。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
azure, python
領域
api, devops
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。