con / con/duct

Handle I/O failures on log files gracefully instead of crashing

Open
#404 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
12
Forks
6
Avg merge
11d 11h
Merged PRs (30d)
1

Description

## Summary

When duct's log files are on NFS with Kerberos authentication (`sec=krb5`), long-running jobs can crash with `PermissionError` when the Kerberos ticket expires. Duct crashes entirely, which can also **kill the monitored command** — in this case a batch orchestration script that had already submitted all SLURM jobs but was still running.

## Root Cause

Investigated on Dartmouth's HPC cluster. NFS mount uses `sec=krb5` with 10-hour ticket lifetime. A ~14-hour duct job outlived the Kerberos ticket, causing all open NFS file handles to return `EACCES` on I/O.

Timeline from a real failure:
- 21:46 — last SSH login refreshed KCM ticket cache (ticket expires ~07:46)
- 07:42 — duct crashes with `PermissionError` (4 min before expected expiry)

The inner SLURM jobs were unaffected because compute nodes use SLURM-managed credentials, not the user's SSH Kerberos ticket.

## Crash Sites

Three threads crash independently with `PermissionError`:

1. **`monitor_process` thread** → `Report.write_subreport()` writing `usage.jsonl`
2. **Two `TailPipe._tail` threads** → `_catch_up()` reading stdout/stderr log files
3. **Main thread** → final `write_subreport()` call after `process.wait()` (fatal crash)

## Impact

- Duct crashes and takes the monitored command with it (child process loses parent / gets SIGPIPE)
- `info.json` left empty (0 bytes), partial data in `usage.jsonl`
- In this case the monitored script had already submitted all SLURM jobs, so compute work was fine — but if the crash had happened earlier, it could have interrupted the orchestration

## Proposed Fix

- On `OSError` in I/O paths: attempt to reopen the file handle and retry (stale handles can be replaced with a fresh `open()`)
- Track file position internally (`self._pos += len(data)`) since `tell()` on a stale handle may also fail
- Cap retries (2–3 attempts); if exhausted, log a warning and skip the write — don't crash
- The monitored command must always keep running; duct should produce whatever summary it can

## Workaround

Keep Kerberos tickets alive during long jobs (e.g., `krenew` or `k5start` in tmux).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.