NVIDIA / NVIDIA/nvbench

Comparison outputs should be colorful when pasted into GitHub

Open
#238 4 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
927
Forks
123
Avg merge
2d 10h
Merged PRs (30d)
2

Description

Copying and pasting a markdown comparison report to GitHub will render as tables and look like this:

T{ct} OffsetT{ct} Elements{io} Ref Time Ref Noise Cmp Time Cmp Noise Diff %Diff Status
I8 I32 2^16 6.092 us 7.97% 6.189 us 9.24% 0.097 us 1.60% SAME
I8 I32 2^20 6.703 us 11.35% 7.095 us 12.43% 0.391 us 5.84% SAME
I8 I32 2^24 14.092 us 2.18% 16.161 us 2.11% 2.068 us 14.68% SLOW
I8 I32 2^28 115.971 us 0.72% 128.828 us 0.24% 12.857 us 11.09% SLOW

Sometimes, pasting as preformatted text is denser:

|  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |      Diff |   %Diff |  Status  |
|---------|---------------|----------------|------------|-------------|------------|-------------|-----------|---------|----------|
|   I8    |      I32      |      2^16      |   6.092 us |       7.97% |   6.189 us |       9.24% |  0.097 us |   1.60% |   SAME   |
|   I8    |      I32      |      2^20      |   6.703 us |      11.35% |   7.095 us |      12.43% |  0.391 us |   5.84% |   SAME   |
|   I8    |      I32      |      2^24      |  14.092 us |       2.18% |  16.161 us |       2.11% |  2.068 us |  14.68% |   SLOW   |
|   I8    |      I32      |      2^28      | 115.971 us |       0.72% | 128.828 us |       0.24% | 12.857 us |  11.09% |   SLOW   |

(a bit wider but less long)

Both versions loose color though, which makes reading the benchmarks on GitHub harder. The console output contained colors. We should find a way to make GitHub render the comparison output with colors as well. One way to do this, is to abuse the diff highlighting and prepend lines with +, -, etc.:

 |  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |      Diff |   %Diff |  Status  |
 |---------|---------------|----------------|------------|-------------|------------|-------------|-----------|---------|----------|
 |   I8    |      I32      |      2^16      |   6.092 us |       7.97% |   6.189 us |       9.24% |  0.097 us |   1.60% |   SAME   |
!|   I8    |      I32      |      2^20      |   6.703 us |      11.35% |   7.095 us |      12.43% |  0.391 us |   5.84% |   SAME   |
-|   I8    |      I32      |      2^24      |  14.092 us |       2.18% |  16.161 us |       2.11% |  2.068 us |  14.68% |   SLOW   |
+|   I8    |      I32      |      2^28      | 115.971 us |       0.72% | 128.828 us |       0.24% | 12.857 us |  11.09% |   SLOW   |

Kudos to @miscco for the idea.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named in the issue. Start by tracing the comparison-report generation and its existing console color handling, then check how pasted output is formatted for GitHub. Done means the comparison remains readable when pasted into GitHub and conveys the status colors described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, markdown
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.