Native bst support on Windows (not WSL)
- Dominant language
- Python
- Stars
- 140
- Forks
- 45
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1021)
In GitLab by [[Gitlab user @aevri]](https://gitlab.com/aevri) on May 9, 2019, 13:54
Windows is a primary development platform for me, I want to be able to use BuildStream on Windows to build binaries for Windows.
Previously, we completed this ticket for remote-only support on WSL: #412
I aired this idea on the mailing list [here](https://mail.gnome.org/archives/buildstream-list/2019-April/msg00057.html), with a TODO list of primary concerns.
## Initial POC
Can compile using host tools, using a 'zero sandbox'.
* [x] !1298: Proof-of-concept of BuildStream building things on Windows.
## Rebased POC, after introduction of `buildbox-casd`, only supporting import elements
Only supports import elements, tested on `doc/examples/first-project/`.
* [x] Experimental branches for support buildbox-casd on Windows, working for 'import-only' workflow: BuildGrid/buildbox/buildbox-common!134, BuildGrid/buildbox/buildbox-casd!89.
* [x] Make a new minimal example of BuildStream working on Windows: https://gitlab.com/BuildStream/buildstream/commits/aevri/win32_minimal
* [x] #1154: Independently confirm the Windows experiment works.
## Official win32 support, using a 'zero sandbox'
Land the changes necessary to be able to run `doc/examples/first-project/` on Windows.
* [x] #1163, !1678: Add CI for native windows support.
* [ ] Full support for [`multiprocessing.set_start_method('spawn')`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method), for platforms that don't support `fork`.
* [ ] !1663: Enable `pytest tests/` with `BST_FORCE_START_METHOD=spawn`
* [ ] !1675: Enable all tests for `BST_FORCE_START_METHOD=spawn`
* [ ] Measure performance of spawn mode, determine if we should be mindful of what references what, so that we're not pickling
more than we need to. This may mean being more explicit in general about the split between the main and worker processes.
* [ ] Change to use some more portable versions of things, re-assess some assumptions, being mindful of performance across platforms. Including:
- Separators: consider `os.sep` **and** `os.altsep`
- Signals, SIGINT, SIGTERM, SIGPIPE not on win32.
- Use of add_child_handler to wait on subprocesses, not available on win32.
- Use of `loop.add_reader(queue._reader.fileno() ...)` (maybe this works? `add_reader` and `_reader` seem to be available on Win32).
- Takes a couple of seconds to spawn a BuildStream process on Windows.
- Can't use UNIX domain sockets to communicate with buildbox-casd on Windows, grpc doesn't have that support, yet.
- ...
- See !1633: WIP: win32: support 'bst show' with buildbox-casd and just enough signal support
* [ ] Support `CASCache.import_directory` in buildbox-casd on Windows.
* [ ] Fully support buildbox-casd on Windows.
* [ ] Introduce SandboxNone, which relies on copying files and good faith. I'd also
like to try this on MacOS, to provide an alternative to the package manager
Homebrew.
* [ ] Bonus: Provide a facility for plugin developers to record and replay job
invocations, for debugging. This works by saving the pickled job data and
launching it again separately.
## Proper sandboxing on Windows
- Evaluate [ProjFS](https://docs.microsoft.com/en-us/windows/desktop/projfs/projected-file-system) on Windows, to provide some filesystem sandboxing.
- Evaluate [Windows Sandbox](https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/Windows-Sandbox/ba-p/301849) for sandboxing.
- Evaluate [Windows Containers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/) for sandboxing.
Contributor guide
Research direction
Start with doc/examples/first-project/ and the outstanding Windows checklist, especially the tests/ runs using BST_FORCE_START_METHOD=spawn. Review the referenced work in #1163, !1663, !1675, and !1633 before choosing a focused task. Done requires the selected Windows workflow or test scope to work without the currently listed platform assumptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100