labgrid-project / labgrid-project/labgrid

File transfer is slow

Open
#1,376 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
528
Forks
278
Avg merge
2d 19h
Merged PRs (30d)
4

Description

When writing an 8MB image to a board I see this:

u-boot-rockchip.bin
      8,896,712 100%    1.03GB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 346,038 bytes  received 35 bytes  692,146.00 bytes/sec
total size is 8,896,712  speedup is 25.71

8774144 bytes (8.8 MB, 8.4 MiB) copied, 10 s, 877 kB/s
17376+1 records in
17376+1 records out
8896712 bytes (8.9 MB, 8.5 MiB) copied, 10.1574 s, 876 kB/s

The last bit seems to be the 'dd'.

If I drop the 'oflag=direct' from USBStorageDriver.write_image I get:

4198912 bytes (4.2 MB, 4.0 MiB) copied, 1 s, 4.2 MB/s
17376+1 records in
17376+1 records out
8896712 bytes (8.9 MB, 8.5 MiB) copied, 2.82611 s, 3.1 MB/s

which is a bit better. Why is direct I/O needed?

commit 27087817f168fbfe5594dd4e2603e336abc05834
Author: Jan Luebbe <jlu@pengutronix.de>
Date:   Thu Jun 18 09:13:27 2020 +0200

    driver/usbstoragedriver: use dd with oflag=direct
    
    This avoid write-caching in the kernel's page cache, reducing disruption
    of concurrent processes and making the progress information more useful.
    
    Remove the leftover debug message of the current working directory.
    
    Signed-off-by: Jan Luebbe <jlu@pengutronix.de>

 labgrid/driver/usbstoragedriver.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

This doesn't make a lot of sense to me. Why not let the kernel handle the caching?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read labgrid/driver/usbstoragedriver.py and commit 27087817f168fbfe5594dd4e2603e336abc05834, then compare the reported transfer behavior with and without dd's oflag=direct. Done means establishing whether direct I/O is required for this driver and documenting the rationale or defining a justified change.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
embedded-iot, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.