files: selinux relabel takes a long time when re-using /home partition

Bug

If I re-use a partition/filesystem for /home/ then the selinux relabel can take a significant amount of time. I suspect that we're relabeling all of the files from the re-used partition and we should probably try to narrow our scope to just files we've touched in Ignition.

Operating System Version

$ rpm-ostree status 
State: idle
Deployments:
● ostree://fedora:fedora/aarch64/coreos/testing-devel
                   Version: 34.20210615.dev.1 (2021-06-15T16:00:07Z)
                    Commit: bc0e38262d392f75d5c65cd9dab8f1da8a72a93c5297b55d726c2ce2c88d5f15
              GPGSignature: (unsigned)

It's a dev build of FCOS since we don't have official aarch64 ones yet.

Ignition Version

ignition-2.10.1-3.fc34.aarch64

Environment

Raspberry Pi 4 (B)

Expected Behavior

Quick relabel during startup.

Actual Behavior

Takes more than 2 minutes for the relabel to happen:

May 25 11:16:22 localhost ignition[670]: INFO     : files: ensureUsers: op(1): [finished] creating or modifying user "core"
May 25 11:16:22 localhost ignition[670]: INFO     : files: ensureUsers: op(2): [started]  adding ssh keys to user "core"
May 25 11:16:22 localhost ignition[670]: wrote ssh authorized keys file for user: core
May 25 11:16:22 localhost ignition[670]: INFO     : files: ensureUsers: op(2): [finished] adding ssh keys to user "core"
May 25 11:16:22 localhost ignition[670]: INFO     : files: createFilesystemsFiles: createFiles: op(3): [started]  writing file "/sysroot/etc/hostname"
May 25 11:16:22 localhost ignition[670]: INFO     : files: createFilesystemsFiles: createFiles: op(3): [finished] writing file "/sysroot/etc/hostname"
May 25 11:16:22 localhost ignition[670]: INFO     : files: createFilesystemsFiles: createFiles: op(4): [started]  writing file "/sysroot/etc/systemd/zram-generator.conf"
May 25 11:16:22 localhost ignition[670]: INFO     : files: createFilesystemsFiles: createFiles: op(4): [finished] writing file "/sysroot/etc/systemd/zram-generator.conf"
May 25 11:16:22 localhost ignition[670]: INFO     : files: createFilesystemsFiles: createFiles: op(5): [started]  writing file "/sysroot/etc/systemd/system/ostree-remount.service.d/before-rfkill.conf"
May 25 11:16:22 localhost ignition[670]: INFO     : files: createFilesystemsFiles: createFiles: op(5): [finished] writing file "/sysroot/etc/systemd/system/ostree-remount.service.d/before-rfkill.conf"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(6): [started]  processing unit "afterburn-sshkeys@core.service"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(6): [finished] processing unit "afterburn-sshkeys@core.service"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(7): [started]  processing unit "var-home.mount"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(7): op(8): [started]  writing unit "var-home.mount" at "/sysroot/etc/systemd/system/var-home.mount"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(7): op(8): [finished] writing unit "var-home.mount" at "/sysroot/etc/systemd/system/var-home.mount"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(7): [finished] processing unit "var-home.mount"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(9): [started]  setting preset to enabled for "afterburn-sshkeys@.service core"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(9): [finished] setting preset to enabled for "afterburn-sshkeys@.service core"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(a): [started]  setting preset to enabled for "var-home.mount"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(a): [finished] setting preset to enabled for "var-home.mount"
May 25 11:16:22 localhost ignition[670]: INFO     : files: op(b): [started]  relabeling 20 patterns
May 25 11:16:22 localhost ignition[670]: DEBUG    : files: op(b): executing: "setfiles" "-vF0" "-r" "/sysroot" "/sysroot/etc/selinux/targeted/contexts/files/file_contexts" "-f" "-"
May 25 11:18:47 localhost ignition[670]: INFO     : files: op(b): [finished] relabeling 20 patterns
May 25 11:18:47 localhost ignition[670]: INFO     : files: files passed
May 25 11:18:47 localhost ignition[670]: INFO     : Ignition finished successfully

Reproduction Steps

Use Ignition config with separate /home/. Here is an example Butane config:

variant: fcos
version: 1.4.0-experimental
storage:
  disks:
    - device: /dev/sda
      wipe_table: false
      partitions:
        - number: 4
          label: root
          size_mib: 51200
          resize: true
        - number: 5
          label: var-home
          size_mib: 102400
  filesystems:
    - path: /var/home
      device: /dev/disk/by-partlabel/var-home
      format: xfs
      wipe_filesystem: false
      label: var-home
      with_mount_unit: true

Install a system place a bunch of files in your home dir. For me I did some rootless podman COSA builds and the contents of my home were about 30G:

[core@rpi4 ~]$ sudo du -sh /home/
30G     /home/

Now re-install with the same Ignition config. Notice the relabel takes much more time.

Start by tracing the files relabel operation that invokes setfiles, using the logged "relabeling 20 patterns" step as the entry point. Reproduce with a reused /var/home filesystem containing many files, then verify that relabeling is limited to paths Ignition touched and that startup no longer scans the entire home partition.