Appending to a file is not idempotent

Bug

Operating System Version

Any

Ignition Version

0.28.0

Environment

Any

Expected Behavior

If storage.files[].append is true, the contents are appended exactly once.

Actual Behavior

In the presence of failures, the contents can be appended multiple times.

Reproduction Steps

On Container Linux:

{
  "ignition": {
    "version": "2.2.0"
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "path": "/foo",
        "append": true,
        "contents": {
          "source": "data:,hello%0A"
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "path": "/bar",
        "append": true,
        "contents": {
          "source": "https://httpbin.org/status/404,404,404,200"
        },
        "mode": 420
      }
    ]
  }
}

Other Information

On Container Linux, if Ignition fails, the machine will automatically reboot after 5 minutes and Ignition will rerun. With the above config, fetching /bar will fail 75% of the time but will eventually succeed, at which point /foo will contain one line per attempt.

We can fix this by appending to files in two passes:

1. Before the first time we append to a particular file in the files stage, check if file.ignition-orig exists. If so, overwrite file with file.ignition-orig before appending to file. If not, copy file to file.ignition-orig.
2. At the end of the files stage, delete the .ignition-orig files.

That approach appropriates *.ignition-orig as private namespace for Ignition. One alternative is to create an .ignition-temp directory at the root of the filesystem and put the backup files there.

The issue names the storage.files append handling and the files stage, but no source files or tests. Start by locating that implementation and reproduce the Container Linux configuration with the failing /bar fetch. Done means /foo receives the contents exactly once across retries, while temporary backup state is cleaned up at the end of the files stage.