[Refactor/Chore] Optimize GitHub Actions CI cost across backend, web, VDB, and Docker lanes

## Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for refactors or chores; if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
- [x] Please do not modify this template :) and fill in all the required fields.

## Description
Continue optimizing GitHub Actions CI cost and feedback time against the current `main` branch without weakening required coverage, merge-queue safety, fork support, or stable required-check reporting.

This issue was reviewed on 2026-07-14 at `d97a29317c`. The remaining work should land as small, independently measurable pull requests rather than one workflow rewrite.

### Already implemented
- [x] Centralized PR and merge-group orchestration in `main-ci.yml`, with concurrency cancellation and duplicate-run suppression.
- [x] Added change detection and stable required-check wrappers for API, CLI, web, full-stack E2E, VDB, and database migration lanes.
- [x] Split API unit and integration tests while preserving merged API coverage reporting.
- [x] Reduced the VDB PR matrix to smoke coverage and moved the full matrix to a weekly/manual workflow.
- [x] Added path-aware Docker image validation and a smaller fork-PR matrix.

### Current evidence
These runs are illustrative snapshots rather than a long-term performance baseline:

- A [Dify Agent dependency-only PR run](https://github.com/langgenius/dify/actions/runs/29294214206) allocated 18 runner-backed jobs even though none of the six main test lanes needed to run. This includes 6 explicit skip jobs, 6 finalizer jobs, 4 style jobs that only discover there is no relevant change after starting, and 2 orchestration jobs.
- A representative [merge-group run](https://github.com/langgenius/dify/actions/runs/29301377052) spent about 12 minutes 42 seconds in the API integration job, followed by about 1 minute 29 seconds in the serialized API coverage job. Full-stack E2E took about 7 minutes.
- Web tests currently use four coverage-enabled shards, followed by a separate report-merging job. The `dify-ui` unit and Storybook suites also start whenever the broad web lane is selected.
- API Python changes can trigger Python Style type checking, Pyrefly Diff, and Pyrefly Type Coverage as separate workflows, with repeated checkout, dependency setup, and base/PR analysis.

### Remaining work

#### P0: Reduce runner-backed no-op work
- Remove the six per-lane `*-skip` jobs; the stable finalizer jobs can determine the no-change result directly from `check-changes`.
- Move style change detection ahead of runner-heavy style jobs so unrelated changes do not start four jobs. In particular, avoid pulling the SuperLinter container before determining whether lintable files changed.
- Re-evaluate whether `pre_job` still provides enough value in addition to workflow concurrency to justify starting a runner for every run. Preserve same-content deduplication and merge-group behavior if it remains.
- Keep the externally visible required-check names and results stable for both changed and unchanged lanes.

#### P0: Consolidate duplicated Pyrefly work
- Share checkout, dependency setup, and base/PR analysis between Pyrefly Diff and Pyrefly Type Coverage wherever their inputs are compatible.
- Reuse artifacts or a common reusable workflow instead of running independent full setup paths.
- Preserve internal-PR and fork-PR comments, failure behavior, and the existing Python Style gate.

#### P1: Shorten the API and full-stack E2E critical path
- Profile API integration time by middleware startup, test collection, and test execution before changing the topology.
- Avoid running the API integration lane for changes that can be proven test-only, documentation-only, or otherwise unrelated to integration behavior.
- Evaluate a smaller PR smoke set versus full merge-group/post-merge E2E only after confirming that required coverage and branch-protection behavior remain equivalent.
- Simplify the serialized API coverage job so it does not install more dependencies than are needed to combine and upload coverage.

#### P1: Reduce web runner minutes
- Benchmark two shards versus four shards using both wall time and total runner minutes; keep four only if the latency benefit justifies the duplicated setup and coverage cost.
- Add narrower path gates for web app tests, `dify-ui` unit tests, and `dify-ui` Storybook tests.
- Measure the cost of coverage on every PR shard and evaluate whether equivalent coverage can be produced with less repeated work.

### Acceptance criteria
- Each optimization PR includes before/after evidence from comparable successful runs, covering both wall time and total runner minutes.
- Required-check names and conclusions remain stable when a lane runs, when it is skipped by path filters, and in merge-group runs.
- Fork PR behavior remains functional and does not gain access to privileged secrets.
- API and web coverage reports remain complete and continue to upload successfully.
- Full VDB and Docker validation coverage is not silently reduced.
- Changes are delivered incrementally so regressions and savings can be attributed to a specific PR.

## Motivation
The original issue led to meaningful improvements, so its old plan no longer accurately described the current repository. The latest workflow still has measurable runner allocation for no-op jobs, duplicated Python analysis, and long API/E2E critical paths.

Keeping this issue open with an updated baseline provides a shared scope for the remaining work while requiring every follow-up PR to demonstrate savings without trading away coverage or reliable required checks.