hpc / hpc/mpifileutils

Use IEC binary prefixes instead of SI units

Open
#345 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
200
Forks
85
Avg merge
3d 21h
Merged PRs (30d)
2

Description

I was comparing dcp with dd for a single file, and was initially puzzled by the performance difference with similar block size.

I then realized that dcp prints GB/s when it is actually GiB/s. When this is accounted for, dd and dcp indeed show the same performance.

In accordance to IEEE 1541-2002 and ISO IEC 80000-13 could we modify the following units to use binary prefixes?
https://github.com/hpc/mpifileutils/blob/57a3578b68d8df0fa53a7662c33c6c21ce72400f/src/common/mfu_util.c#L310-L322

This is confusing because other I/O tools use the right unit and print both types of units to show that they are aware of the potential ambiguity:
dd:

419430400 bytes (419 MB, 400 MiB) copied, 0.0436056 s, 9.6 GB/s

See https://github.com/coreutils/coreutils/blob/8d13292a73ecf1f265f77731d3ace29866e3d616/src/dd.c#L588-L591

fio:

   READ: bw=506MiB/s (530MB/s), 506MiB/s-506MiB/s (530MB/s-530MB/s), io=2048MiB (2147MB), run=4049-4049msec

See https://fio.readthedocs.io/en/latest/fio_doc.html#units

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the unit-formatting code in src/common/mfu_util.c around lines 310-322, then reproduce dcp's single-file throughput output for comparison. Done means the displayed rates use IEC binary prefixes consistently and no longer label GiB/s values as GB/s.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
cli, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.