LinearTapeFileSystem / LinearTapeFileSystem/ltfs

How should upper-layer applications handle permanent write errors and extent cleanup?

Open
#647 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
352
Forks
110
Avg merge
2h 50m
Merged PRs (30d)
2

Description

I performed a test in which I wrote four files to a tape using four parallel threads. The four threads use a lock around the `write()` system call to ensure that the drive is accessed serially. However, operations such as `open()`, `fsync()`, and `close()` for each file are not protected by the lock.

After all four jobs completed, I called `ltfs.sync()` once.

During the writing of the second file, a "Cannot write block" error occurred. The tape entered a permanent write error state, and LTFS immediately performed an extent cleanup. The relevant LTFS log is as follows:

```text
Sep 15 01:54:53
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 64 LTFS11337I Update index-dirty
flag (1) - NO_BARCODE (0x0x55c5635ea710).

Sep 15 01:55:29
vdm3000-udm-utdmagent-0 user.err ltfs[98]: 65 LTFS12045E Cannot write block:
backend call failed (-20301). Dropping to read-only mode.

Sep 15 01:55:29
vdm3000-udm-utdmagent-0 user.err ltfs[98]: 65 LTFS11072E Cannot write blocks:
failed to write to the medium (-20301).

Sep 15 01:55:29
vdm3000-udm-utdmagent-0 user.err ltfs[98]: 65 LTFS11077E Cannot write: failed
to write blocks to the medium (-20301).

Sep 15 01:55:29
vdm3000-udm-utdmagent-0 user.warn ltfs[98]: 65 LTFS13014W Data partition
writer: failed to write data to the tape (-20301).

Sep 15 01:55:29
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS13024I Clean up extents and
append index at index partition (-20301).

Sep 15 01:55:31
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS17292I Current position is
(1, 10246), Error position is (1, 10117).

Sep 15 01:55:31
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS13025I Truncate extents
larger than position (1, 10117), block size = 524288.

Sep 15 01:55:31
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS11334I Remove extent :
utdm.0.0.1613966600.2147720233 (9407, 439877632).

Sep 15 01:55:31
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS11343I Try to write an index
on the IP on NO_BARCODE because of a permanent write error on the DP..

Sep 15 01:55:49
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS17235I Writing index of
NO_BARCODE to a (Reason: Write perm, 5 files) 11C9D0D0D3.

Sep 15 01:55:58
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS17236I Wrote index of
NO_BARCODE (Gen = 2, Part = a, Pos = 8, 11C9D0D0D3).

Sep 15 01:55:58
vdm3000-udm-utdmagent-0 user.info ltfs[98]: 65 LTFS11337I Update index-dirty
flag (0) - NO_BARCODE (0x0x55c5635ea710).

Sep 15 01:55:58
vdm3000-udm-utdmagent-0 user.err ltfs[98]: 8c LTFS14022E Cannot flush file data
to the medium. Data might be lost (ltfs_fuse_flush).
```

On the application side, I observed that the `write()` calls for the second, third, and fourth files returned `errno = 2`. Then `ltfs.sync()` returned `-1`.

In this situation, LTFS performed an extent cleanup after the permanent write error. My questions are:

1. **Can the contents of the first file, for which some `write()` calls had already returned success, be partially or completely affected by the cleanup?**

In other words, is it possible that some data from the first file that had previously been reported as successfully written by `write()` was truncated or removed because of the subsequent permanent write error?

2. **How can an upper-layer application reliably detect that the tape has entered the permanent write error ("write-perm") state?**

Is there an LTFS API, ioctl, status interface, return code, or other mechanism that an application can use to detect this state without parsing the LTFS log?

3. **How can the upper-layer application determine that LTFS has performed an extent cleanup or rollback?**

Is there any API, event, status, or other mechanism that exposes this information to the application?

4. **What should the upper-layer application do after `ltfs.sync()` returns `-1`?**

What checks are recommended to determine whether previously successful writes are still valid and whether any data may have been lost or truncated?

5. More generally, **what guarantees does LTFS provide for a successful `write()` in the presence of a subsequent permanent write error and extent cleanup?**

6. It is quite possible that the "Cannot write block" error was caused by a drive-related problem. In that case, how can we reliably determine whether the root cause was the drive itself, rather than the tape medium or the I/O path (e.g., HBA/FC/backend)?
I would like to understand the expected semantics in this situation and the recommended way for an upper-layer application to detect and safely handle a permanent write error and the resulting extent cleanup.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the application-visible behavior of write(), fsync(), close(), and ltfs.sync() against the permanent write error and extent-cleanup log entries in the issue. Done means the semantics of successful writes, cleanup, error detection, and recommended handling after ltfs.sync() returns -1 are defined, including how drive, medium, and I/O-path causes can be distinguished.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
backend, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.