aai-institute / aai-institute/nnbench

Support greedy / immediate result reporting

未關閉
#158 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
21
分支
4
PR 合併指標
PR 指標待擷取

描述

Instead of accumulating all results and then passing them off to a reporter like we do now, it is sometimes beneficial to report results as soon as they are available (for example in a console).

This has some implications:

* First of all, the runner requires a slot for a `BenchmarkReporter` instance in its run method.
* Secondly, for anything other than a console reporter, this could incur higher overhead than just a batched reporting, e.g. over a connection (HTTP, streaming, file writing). Maybe a "greedy" reporting could be a "batched reporting with batch size = 1" scenario.
* Thirdly, for anything table-based, any non-static ("unbounded" in Google Dataflow lingo) data stream requires certain parameters up-front for correct rendering (e.g. max column width). These either have to be inferred or set upfront by some heuristic.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。