aai-institute / aai-institute/nnbench
Support greedy / immediate result reporting
- 主要語言
- Python
- 星號
- 21
- 分支
- 4
- PR 合併指標
- PR 指標待擷取
描述
Instead of accumulating all results and then passing them off to a reporter like we do now, it is sometimes beneficial to report results as soon as they are available (for example in a console).
This has some implications:
* First of all, the runner requires a slot for a `BenchmarkReporter` instance in its run method.
* Secondly, for anything other than a console reporter, this could incur higher overhead than just a batched reporting, e.g. over a connection (HTTP, streaming, file writing). Maybe a "greedy" reporting could be a "batched reporting with batch size = 1" scenario.
* Thirdly, for anything table-based, any non-static ("unbounded" in Google Dataflow lingo) data stream requires certain parameters up-front for correct rendering (e.g. max column width). These either have to be inferred or set upfront by some heuristic.
貢獻指南
評估
這個 Issue 還沒有評估資料。