azure train jobs fail randomely when accessing blobs with azureml fsspec

I have a tensorflow training pipeline that randomely fails when reading image-blobs from azure file storage. Sometimes the training finishes successfully, sometimes it barely reaches 50 epochs. However, the reason is always the same when it fails.

The training pipeline loads images using azure-ml fsspec, so that I can use my pipeline with online and offline data without having the need to change my code.

Reading the blob is done with (path is a tensorflow-tensor):

path_str = path.numpy().decode("utf-8")

try:

with self._fs.open(path_str, "rb") as f:

return f.read()

except Exception as e:

self._model.stop_training = True

print(f"Failed loading file: {path_str}, error: {e}")

sys.exit(1) # Non-zero exit code signals failure to Azure ML

The function usually works correctly. I've been using it across different projects and have finished trainings in many cases.

The error is usually indicated like that in the log-file:
---------------------
root
root
root
root
root
0
1
2
20
22
24
26
29
3
31
33
35
38
4
5
50
54
6
Name: cat
Umask: 0022
State: R (running)
Tgid: 1419581
Ngid: 0
Pid: 1419581
PPid: 14
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 64
Groups: 0
NStgid: 1419581
NSpid: 1419581
NSpgid: 14
NSsid: 14
Kthread: 0
VmPeak: 3364 kB
VmSize: 3364 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 1664 kB
VmRSS: 1664 kB
RssAnon: 0 kB
RssFile: 1664 kB
RssShmem: 0 kB
VmData: 360 kB
VmStk: 144 kB
VmExe: 16 kB
VmLib: 1796 kB
VmPTE: 48 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
THP_enabled: 1
untag_mask: 0xffffffffffffffff
Threads: 1
SigQ: 0/112242
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000181000000
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Seccomp_filters: 0
Speculation_Store_Bypass: thread vulnerable
SpeculationIndirectBranch: always enabled
Cpus_allowed: f
Cpus_allowed_list: 0-3
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 2
nonvoluntary_ctxt_switches: 0
x86_Thread_features:
x86_Thread_features_locked:
thread '' panicked at 'Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', rslex-core/src/lib.rs:128:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-10-28 16:54:52.690564: W tensorflow/core/framework/op_kernel.cc:1827] UNKNOWN: InvalidArgumentError: Failed loading file: , error: __enter__
Traceback (most recent call last):

File "/mnt/azureml/cr/j/03d1124a2e84448f92b9e479a0092c16/exe/wd/modules/core.py", line 362, in _read_stream
with self._fs.open(path_str, "rb") as f:

AttributeError: __enter__

------------------------

Afterwards it is repeating the "root" statement in the log file.

Is there anything I can do to prevent this from happening? It is really annoying that training runs don't finish.

Thanks.