apache / apache/texera

CI has no Windows coverage: hadoop dependency bumps can silently break Windows development

Open
#6,503 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Add a small Windows CI leg that exercises the OS-sensitive paths, so dependency bumps that break Windows development fail in the PR instead of on a dev machine weeks later.

**Motivation:** #6487 was a Windows-only regression that CI could not catch: the hadoop-common bumps #6201 (3.3.1 → 3.4.3) and #6227 (→ 3.5.0) made every local Iceberg write require `winutils.exe`, breaking all workflow executions on Windows dev machines — while `ubuntu-latest` and `macos-latest` CI stayed green. See the discussion in #6488.

A minimal starting point:

| | |
|---|---|
| Runner | `windows-latest` |
| Scope | `WorkflowCore` unit tests (or just the file-system-sensitive specs, e.g. `IcebergUtilSpec` + `WinutilsFreeLocalFileSystemSpec`) |
| Would it have caught #6487? | Yes — the regression test added in #6488 fails on a stock Windows runner without the fix |

Notes / open questions:

- A full `build / amber` Windows leg is likely too heavy (Windows runners are slower and billed at 2x; amber integration tests spawn Python workers via `signal.SIGKILL`, which does not exist on Windows), so scoping to unit tests of the storage/file-system layer keeps the cost low.
- A Windows leg would also cover the Windows-only branches that Linux-based Codecov runs cannot execute.
- Could run on `main` + dependency-labeled PRs only, if PR latency is a concern.

### Task Type

- [ ] Refactor / Cleanup
- [x] DevOps / Deployment / CI
- [x] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing CI workflow and the WorkflowCore unit-test configuration, then run the file-system-sensitive IcebergUtilSpec and WinutilsFreeLocalFileSystemSpec on a Windows runner. Done means a lightweight windows-latest CI leg runs the selected tests reliably and catches the regression described in #6488 without attempting the incompatible full amber suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, scala
Domain
ci-cd, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.