BuildStream 2 artifacts lose the unix write permission bit for the user (`u+w`)
- Dominant language
- Python
- Stars
- 140
- Forks
- 45
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
## Context
As far as I understand, BuildStream is supposed to preserve unix file permissions to a certain extent. However, the user write permission bit seems to be lost from files when checking out built artifacts. This issue is not seen with bst16.
## Description
When checking out artifacts using `--hardlinks` in bst2, it seems that the `u+w` permission is removed from every file (or almost every file?) compared to the permission for that file when it was originally collected as part of `bst build`.
From limited testing with the [Flatpak repo](https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/9b70e597426135ae4b8631f80c1819677656271e/elements/flatpak-release.bst) built in the `freedesktop-sdk` project, it appears that only the user write bit disapears, while all other unix permission bits are retained.
Slightly unrelated to this, and potentially a different issue is when checking out artifacts _without_ `--hardlinks`, bst2 appears to set file permissions according to `umask`, resulting in file permissions which are yet again different from the file permissions that were present when the files were originally collected.
## Impact
Among other issues, this behavior leads OSTree repos which are collected into artifacts to be considered corrupted by `ostree fsck`. This is because OSTree repo include file permissions in their hash calculaition for objects, so this file permission change is detected as a file a corruption.
This file permission change and its effect on OSTree repos was found to be reason why the `freedesktop-sdk` project couldn't publish its Flatpak repo to its remote release server since moving from bst16 to bst2 in the `master` branch.
The only workaround found for this issue was to check out artifacts without `--hardlinks` and with a specific `umask` that provides the expected file permissions. However, this results in massive disk I/O overhead for the `freedesktop-sdk` project.
## References:
- The OSTree error appears when publishing in the `freedesktop-sdk` CI: (this is from the push job of the bst2 MR, where the issue was first seen)
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/2929504305#L46
- The OSTree error similarly appears when using the local `ostree fsck` command here:
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/3315563117#L1249
cc: @abderrahim, @gtristan, @nanonyme
Contributor guide
Research direction
Start by reproducing an artifact checkout with BuildStream 2 using --hardlinks, then compare file permissions with those captured during bst build and with a checkout using the default mode. The fix is done when checkout preserves the user write bit and other permissions without relying on a special umask, and the resulting OSTree repository passes ostree fsck.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100