workload yaml files with hard coded endpoint_url resulting in DLIO workloads to S3 storage fail

It will be good to propagate/set the AWS_ENDPOINT_URL values (being set in mlp-storage/.env) to the endpoint_url value in dlio/workload/*.yaml .

Currently, the endpoint_url: is hard-coded to static value of https://172.16.1.40:9000 which results in S3 failures. Setting this endpoint_url: value to the correct value based on the deployed S3 storage results in successful run.

Also, it will be good to specify the data_folder: as well as checkpoint_folder: via mlp-storage/.env or as option in mlpstorage training datagen/run --objects --bucket-name <bucket-value> / mlpstorage checkpointing run --objects --bucket-name <bucket-value>.

FYI, @russfellows @FileSystemGuy

Issue:

• Currently, the endpoint_url: is hard-coded to static value of https://172.16.1.40:9000 which results in S3 failures.

$ grep endpoint_url *.yaml
datagen_s3dlio_s3.yaml:    endpoint_url: http://localhost:9000
llama3_8b_checkpoint_minio.yaml:    endpoint_url: https://172.16.1.40:9000
llama3_8b_checkpoint_s3torch.yaml:    endpoint_url: https://172.16.1.40:9000
pytorch_s3dlio_azure.yaml:    # endpoint_url: https://mystorageaccount.blob.core.windows.net
pytorch_s3dlio_local_test.yaml:    endpoint_url: http://localhost:9000
pytorch_s3dlio.yaml:    endpoint_url: http://localhost:9000
pytorch_s3dlio.yaml:  # For MULTIPLE endpoints, replace endpoint_url with endpoint_uris (s3dlio only):
pytorch_s3torchconnector.yaml:    endpoint_url: http://localhost:9000
unet3d_h100_minio_datagen.yaml:    endpoint_url: https://172.16.1.40:9000
unet3d_h100_minio.yaml:    endpoint_url: https://172.16.1.40:9000
unet3d_h100_s3dlio_datagen.yaml:    endpoint_url: https://172.16.1.40:9000
unet3d_h100_s3dlio.yaml:    endpoint_url: https://172.16.1.40:9000

$ CHECKPOINTS=1 NP=2 bash tests/object-store/dlio_s3dlio_checkpoint.sh
[env] Loading credentials from .env

════════════════════════════════════════════════════════
  DLIO Checkpoint — s3dlio + MinIO  (llama3-8b)
════════════════════════════════════════════════════════
  Bucket      : chckpt-test1
  Objects at  : s3://chckpt-test1/s3dlio/llama3-8b/
  Endpoint    : https://192.168.7.124:8111
  MPI ranks   : 2   (default=1; full run: NP=8 bash tests/object-store/dlio_s3dlio_checkpoint.sh)
  Checkpoints : 1 write + 1 read
  Per-rank    : ~13.1 GB per checkpoint  (ZeRO-3, 8 ranks)
  Run dir     : /tmp/dlio-s3dlio-checkpoint-20260408_194513
════════════════════════════════════════════════════════
Checking bucket reachability: s3://chckpt-test1/ ...
  Bucket accessible — 0 top-level entries
..
..
--------------------------------------------------------------------------
[DEBUG DLIOBenchmark.__init__] After LoadConfig:
  storage_type   = <StorageType.S3: 's3'>
  storage_root   = 'chckpt-test1'
  **storage_options= {'endpoint_url': 'https://172.16.1.40:9000', 'region': 'us-east-1', 's3_force_path_style': True, 'storage_library': 's3dlio'} <-- HARDCODED WRONG ENDPOINT URL from llama3_8b_checkpoint_s3dlio.yaml:** 
  data_folder    = './data/'
  framework      = <FrameworkType.PYTORCH: 'pytorch'>
  num_files_train= 8
  record_length  = 65536
  generate_data  = False
  do_train       = False
  do_checkpoint  = True
  epochs         = 1
  batch_size     = 1
..
..
S3DLIOWriter] Using MultipartUploadWriter (single endpoint)
[S3DLIOWriter]   part_size=128 MB, max_in_flight=8
[Writer] ERROR: Failed to create storage writer: Multipart init failed: CreateMultipartUpload failed
[Writer] ERROR: Failed to create storage writer: Multipart init failed: CreateMultipartUpload failed

Interim Solution:

Correct the endpoint_url

$ diff llama3_8b_checkpoint_s3dlio.yaml /tmp/kums/llama3_8b_checkpoint_s3dlio.yaml.orig 
75c75
<     endpoint_url: https://192.168.7.124:8111
---
>     endpoint_url: https://172.16.1.40:9000

$ CHECKPOINTS=1 NP=2 bash tests/object-store/dlio_s3dlio_checkpoint.sh
[env] Loading credentials from .env

════════════════════════════════════════════════════════
  DLIO Checkpoint — s3dlio + MinIO  (llama3-8b)
════════════════════════════════════════════════════════
  Bucket      : chckpt-test1
  Objects at  : s3://chckpt-test1/s3dlio/llama3-8b/
  Endpoint    : https://192.168.7.124:8111
  MPI ranks   : 2   (default=1; full run: NP=8 bash tests/object-store/dlio_s3dlio_checkpoint.sh)
  Checkpoints : 1 write + 1 read
  Per-rank    : ~13.1 GB per checkpoint  (ZeRO-3, 8 ranks)
  Run dir     : /tmp/dlio-s3dlio-checkpoint-20260408_195249
════════════════════════════════════════════════════════

Checking bucket reachability: s3://chckpt-test1/ ...
  Bucket accessible — 0 top-level entries
..
[DEBUG DLIOBenchmark.__init__] After LoadConfig:
  storage_type   = <StorageType.S3: 's3'>
  storage_root   = 'chckpt-test1'
  storage_options= {'endpoint_url': 'https://192.168.7.124:8111', 'region': 'us-east-1', 's3_force_path_style': True, 'storage_library': 's3dlio'}
  data_folder    = './data/'
  framework      = <FrameworkType.PYTORCH: 'pytorch'>
  num_files_train= 8
  record_length  = 65536
  generate_data  = False
  do_train       = False
  do_checkpoint  = True
  epochs         = 1
  batch_size     = 1
..
..
================================================================================
LOAD RESULTS
================================================================================
I/O:        35.2792s @ 1.27 GB/s
Total:      35.3952s
Chunks:     360
================================================================================
[OUTPUT] 2026-04-08T19:55:04.702577 Loaded optimizer checkpoint in 35.3953 seconds
[OUTPUT] 2026-04-08T19:55:04.702783 Finished loading checkpoint 1 for epoch 1 in 41.9517 s; Throughput: 2.4958 GB/s
[OUTPUT] 2026-04-08T19:55:04.704114 Checkpointing write started

================================================================================
LOAD RESULTS
================================================================================
I/O:        35.4110s @ 1.27 GB/s
Total:      35.6388s
Chunks:     360
================================================================================
[OUTPUT] 2026-04-08T19:55:04.983969 Saved outputs in /tmp/dlio-s3dlio-checkpoint-20260408_195249
[OUTPUT] Averaged metric over all steps/epochs
[METRIC] ==========================================================
[METRIC] Number of Simulated Accelerators: 2 
[METRIC] Checkpoint save duration (seconds): 78.6040 (0.0000)
[METRIC] Checkpoint save I/O Throughput (GB/second): 1.3320 (0.0000)
[METRIC] Checkpoint load duration (seconds): 41.9517 (0.0000)
[METRIC] Checkpoint load I/O Throughput (GB/second): 2.4958 (0.0000)
[METRIC] ==========================================================

[OUTPUT] 2026-04-08T19:55:04.984681 outputs saved in RANKID_output.json

✅  Checkpoint test complete — results in /tmp/dlio-s3dlio-checkpoint-20260408_195249

We can further verify dlio-s3dlio-checkpoint dataset in the S3 storage with above interim solution

$ aws --ca-bundle=ca.pem --endpoint-url=$S3URL_HSE s3 ls s3://chckpt-test1/s3dlio/llama3-8b/global_epoch1_step1/
2026-04-08 19:53:10 8030261248 zero_pp_rank_0_mp_rank_0_model_states.pt
2026-04-08 19:54:17 48181813248 zero_pp_rank_0_mp_rank_0_optim_states.pt
2026-04-08 19:53:10 8030261248 zero_pp_rank_1_mp_rank_0_model_states.pt
2026-04-08 19:54:17 48181813248 zero_pp_rank_1_mp_rank_0_optim_states.pt

Start with the endpoint_url entries in dlio/workload/*.yaml and the AWS_ENDPOINT_URL values in mlp-storage/.env. Review tests/object-store/dlio_s3dlio_checkpoint.sh and the mlpstorage training datagen/run and checkpointing run commands. Done means workloads use the deployed S3 endpoint and data_folder and checkpoint_folder can be configured as requested, with the checkpoint test succeeding.