apache / apache/buildstream

BuildStream 2 artifacts lose the unix write permission bit for the user (`u+w`)

Open
#1,790 8 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

## Context
As far as I understand, BuildStream is supposed to preserve unix file permissions to a certain extent. However, the user write permission bit seems to be lost from files when checking out built artifacts. This issue is not seen with bst16.

## Description
When checking out artifacts using `--hardlinks` in bst2, it seems that the `u+w` permission is removed from every file (or almost every file?) compared to the permission for that file when it was originally collected as part of `bst build`.

From limited testing with the [Flatpak repo](https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/9b70e597426135ae4b8631f80c1819677656271e/elements/flatpak-release.bst) built in the `freedesktop-sdk` project, it appears that only the user write bit disapears, while all other unix permission bits are retained.

Slightly unrelated to this, and potentially a different issue is when checking out artifacts _without_ `--hardlinks`, bst2 appears to set file permissions according to `umask`, resulting in file permissions which are yet again different from the file permissions that were present when the files were originally collected.

## Impact
Among other issues, this behavior leads OSTree repos which are collected into artifacts to be considered corrupted by `ostree fsck`. This is because OSTree repo include file permissions in their hash calculaition for objects, so this file permission change is detected as a file a corruption.

This file permission change and its effect on OSTree repos was found to be reason why the `freedesktop-sdk` project couldn't publish its Flatpak repo to its remote release server since moving from bst16 to bst2 in the `master` branch.

The only workaround found for this issue was to check out artifacts without `--hardlinks` and with a specific `umask` that provides the expected file permissions. However, this results in massive disk I/O overhead for the `freedesktop-sdk` project.

## References:
- The OSTree error appears when publishing in the `freedesktop-sdk` CI: (this is from the push job of the bst2 MR, where the issue was first seen)
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/2929504305#L46
- The OSTree error similarly appears when using the local `ostree fsck` command here:
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/3315563117#L1249

cc: @abderrahim, @gtristan, @nanonyme

Contributor guide

Open the contributing guide

Research direction

Start by reproducing an artifact checkout with BuildStream 2 using --hardlinks, then compare file permissions with those captured during bst build and with a checkout using the default mode. The fix is done when checkout preserves the user write bit and other permissions without relying on a special umask, and the resulting OSTree repository passes ostree fsck.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.