[e2eTESTING] V tests: Kerchunk vs Pyfive engines

Local tests on V Computer

• network: UoR LAN/eth0 (not over WiFi)
• CPU:

     *-cpu
          product: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
          vendor: Intel Corp.
          physical id: 1
          bus info: cpu@0
          size: 2303MHz
          capacity: 2800MHz
          width: 64 bits

Test code:

import os
import numpy as np


from activestorage.active import Active


S3_ACTIVE_URL_Bryan = "https://192.171.169.248:8080"
S3_BUCKET = "bnl"

def gold_test():
    """Run somewhat as the 'gold' test."""
    storage_options = {
        'key': "f2d55c6dcfc7618b2c34e00b58df3cef",
        'secret': "$/'#M{0{/4rVhp%n^(XeX$q@y#&(NM3W1->~N.Q6VP.5[@bLpi='nt]AfH)>78pT",
        'client_kwargs': {'endpoint_url': "https://uor-aces-o.s3-ext.jc.rl.ac.uk"},
    }
    active_storage_url = "https://192.171.169.248:8080"
    bigger_file = "ch330a.pc19790301-bnl.nc"

    test_file_uri = os.path.join(
        S3_BUCKET,
        bigger_file
    )
    print("S3 Test file path:", test_file_uri)
    active = Active(test_file_uri, 'UM_m01s16i202_vn1106', storage_type="s3",
                    storage_options=storage_options,
                    active_storage_url=active_storage_url)
    # old test with 3GB file
    # active2 = Active(test_file_uri, 'm01s06i247_4', storage_type="s3",
    #                 storage_options=storage_options,
    #                 active_storage_url=active_storage_url)

    active._version = 1
    active._method = "min"

    result = active[:]
    # result = active[0:3, 4:6, 7:9]  # standardized slice

    print("Result is", result)
    return result

Kerchunk is restricted to Dataset of interest:

Looking only at a single Dataset <HDF5 dataset "UM_m01s16i202_vn1106": shape (40, 1920, 2560), type "<f4">

Chunks

Both Kerchunk and Pyfive send variable (give or take 5 or 10) numbers of chunks to Reductionist; order of magnitude is 3360 chunks.

Kerchunk-based Pipeline

Result is 4677.8594 (stable)

• 18.03user 2.24system 1:37.77elapsed 20%CPU (0avgtext+0avgdata 202112maxresident)k
• 20.00user 2.02system 1:35.60elapsed 23%CPU (0avgtext+0avgdata 203124maxresident)k
• 19.64user 2.26system 1:34.86elapsed 23%CPU (0avgtext+0avgdata 201880maxresident)k
• 20.95user 2.43system 1:34.75elapsed 24%CPU (0avgtext+0avgdata 200884maxresident)k
• 14.94user 1.49system 1:34.19elapsed 17%CPU (0avgtext+0avgdata 201932maxresident)k
• 15.47user 1.72system 1:47.83elapsed 15%CPU (0avgtext+0avgdata 203052maxresident)k
• 20.04user 2.19system 1:33.50elapsed 23%CPU (0avgtext+0avgdata 202192maxresident)k
• 19.73user 2.08system 1:35.95elapsed 22%CPU (0avgtext+0avgdata 202144maxresident)k
• 20.65user 2.44system 1:31.98elapsed 25%CPU (0avgtext+0avgdata 200952maxresident)k

Kerchunk indexing and JSON file writing times:

• Time to Kerchunk and write JSON file 21.811710596084595
• Time to Kerchunk and write JSON file 20.934044361114502
• Time to Kerchunk and write JSON file 21.715813636779785
• Time to Kerchunk and write JSON file 21.793660879135132

Pyfive-based pipeline

Result is 4677.8594 (stable)

• 21.54user 3.07system 1:22.10elapsed 29%CPU (0avgtext+0avgdata 195224maxresident)k
• 21.28user 2.79system 1:19.94elapsed 30%CPU (0avgtext+0avgdata 196944maxresident)k
• 21.47user 2.73system 1:25.87elapsed 28%CPU (0avgtext+0avgdata 198084maxresident)k
• 21.05user 2.93system 1:35.86elapsed 25%CPU (0avgtext+0avgdata 197568maxresident)k
• 21.45user 2.78system 1:30.15elapsed 26%CPU (0avgtext+0avgdata 197820maxresident)k

Sliced Kerchunk (slice [0:3, 4:6, 7:9])

• Time to Kerchunk and write JSON file 21.60s; 27s TOTAL
• Time to Kerchunk and write JSON file 22.16s; 27s TOTAL
• Time to Kerchunk and write JSON file 21.15s; 27s TOTAL
• Time to Kerchunk and write JSON file 22.61s; 28s TOTAL

Sliced Pyfive (slice [0:3, 4:6, 7:9])

• 14s TOTAL
• 13s TOTAL
• 12s TOTAL
• 13.4s TOTAL

Start with the supplied Python test code and the gold_test() entry point, then reproduce the Kerchunk and Pyfive pipelines and their reported slices. Done means establishing whether the result and timing differences are reproducible and documenting the comparison or a specific follow-up.