[Refactor/Chore] Migrate handwritten web API helpers to generated contracts
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
## Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for refactors or chores; if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
## Description
The web frontend still has broad handwritten REST usage through `web/service/base.ts`, even though production API contracts are now available via `@dify/contracts` and `web/service/client.ts` (`consoleClient` / `consoleQuery`).
A local scan of production TypeScript/TSX files found:
- `460` calls to `web/service/base.ts` exported helpers across `62` files.
- `411` ordinary JSON REST calls (`get`, `post`, `put`, `del`, `patch`) across `43` files.
- `47` special calls for SSE, upload, public API, or marketplace API across `23` files.
- `2` utility calls (`request` / `handleStream`).
For the `411` ordinary JSON REST calls, a rough method/path match against the generated console + enterprise contracts found:
- `336` calls directly match generated contracts.
- `66` calls use dynamic URL wrappers and need call-site-level migration.
- `8` calls appear to be missing from the available contracts.
- `1` call partially matches contracts.
High-volume handwritten API files include:
- `web/service/common.ts`: 44 ordinary JSON calls.
- `web/service/datasets.ts`: 32 ordinary JSON calls.
- `web/service/use-plugins.ts`: 27 ordinary JSON calls.
- `web/service/apps.ts`: 25 ordinary JSON calls.
- `web/service/use-pipeline.ts`: 21 ordinary JSON calls.
- `web/service/tools.ts`, `web/service/use-common.ts`, `web/service/use-tools.ts`: 20 ordinary JSON calls each.
Suggested migration scope:
1. Migrate direct, non-streaming console JSON endpoints from `get` / `post` / `put` / `del` / `patch` to `consoleClient` / `consoleQuery`.
2. Start with fully matched, non-dynamic service modules such as access control, knowledge, billing, `use-pipeline`, and `use-tools`.
3. Treat dynamic URL wrappers separately. Examples include generic helpers in `web/service/common.ts`, `web/service/apps.ts`, `web/service/log.ts`, and `web/service/use-plugins-auth.ts`; these should be migrated from the concrete call sites instead of blindly replacing the wrapper internals.
4. Keep `ssePost`, `sseGet`, `upload`, public API helpers, and marketplace helpers for now unless their runtime behavior is explicitly supported by generated clients.
Contract gaps or unresolved paths from the scan:
- `POST /setup`
- `POST /init`
- `GET /init`
- `GET /setup`
- `GET /version`
- `GET /workspaces/current/tool-provider/model/tools`
- `GET /workspaces/current/tool-provider/builtin/{provider}/credentials_schema`
- `GET /auth/plugin/datasource/{pluginId}/{provider}`
Partial contract coverage:
- `GET /apps/{app_id}/workflows/draft/conversation-variables` exists.
- `GET /snippets/{snippet_id}/workflows/draft/conversation-variables` exists.
- `GET /rag/pipelines/{pipeline_id}/workflows/draft/conversation-variables` did not match in the generated contract scan.
## Motivation
The `get`, `post`, `put`, `del`, and `patch` helpers in `web/service/base.ts` are already marked as deprecated for console JSON APIs. Migrating contract-covered endpoints to generated clients should improve:
- Request and response type safety.
- Endpoint drift detection.
- TanStack Query key consistency.
- Reuse of generated contract input/output types.
- Long-term maintainability by reducing handwritten REST helpers.
This should not be a blanket replacement. Streaming, upload, binary, public webapp, and marketplace flows currently rely on behavior in `web/service/base.ts` such as SSE parsing, abort handling, progress handling, webapp passport headers, and marketplace-specific headers. The existing `packages/contracts/non-json-openapi-responses.md` also documents special streaming and binary response behavior that should be considered before migrating those paths.
## Additional Context
Suggested first phase:
- Migrate the `336` directly matched ordinary JSON call sites.
- Leave dynamic URL wrappers and contract gaps as follow-up work.
- Add focused tests around migrated service hooks where behavior, query invalidation, or payload shapes are non-trivial.
Suggested follow-up phase:
- Add or repair contracts for the unmatched endpoints listed above, or explicitly document why they should remain handwritten.
- Refactor dynamic URL wrapper APIs so concrete endpoints are expressed at the call sites and can use generated clients safely.
## Proposed PR Breakdown
This migration should be split by review boundary and runtime risk rather than by raw file count.
Recommended PR groups:
1. **Low-risk direct contract migrations**
- Scope: static endpoints, ordinary JSON responses, no dynamic URL argument, no SSE, no upload.
- Good first migration PR to establish the pattern.
- Candidate modules: `billing`, `use-education`, `strategy`, `use-workspace`, and small fully matched access-control helpers.
2. **Access control / RBAC**
- Scope: `web/service/access-control/*`.
- Reason: clear domain boundary and the codebase already uses some `consoleClient` / `consoleQuery` in this area.
3. **Knowledge / dataset APIs**
- Suggested split:
- Dataset list/detail/settings: `datasets.ts`, `knowledge/use-dataset.ts`.
- Documents, segments, metadata: `knowledge/use-document.ts`, `knowledge/use-segment.ts`, `knowledge/use-metadata.ts`.
- Create/import/hit-testing: `knowledge/use-create-dataset.ts`, `knowledge/use-import.ts`, `knowledge/use-hit-testing.ts`.
4. **Apps / app configuration**
- Scope: `apps.ts`, but split by feature when needed.
- Suggested boundaries: CRUD/copy/import/export, site config/API keys, tracing/webhook.
- Reason: app service changes touch user-facing flows and should stay reviewable.
5. **Workflow non-streaming JSON APIs**
- Scope: non-SSE endpoints in `workflow.ts` and `use-workflow.ts`.
- Explicitly exclude `ssePost`, `sseGet`, `handleStream`, workflow run streams, and resume streams.
6. **Plugins / tools / model provider APIs**
- Suggested split:
- Tool and tool-provider APIs.
- Plugin install/task/list APIs.
- Model-provider APIs.
- Plugin auth dynamic URL APIs as a later PR.
- Keep marketplace APIs out of the first console-contract migration PRs.
7. **Dynamic URL wrappers**
- Scope: wrappers that accept `url`, `urlPath`, or `params.url`, such as parts of `common.ts`, `apps.ts`, `log.ts`, `use-plugins-auth.ts`, and workflow helpers.
- Approach: migrate from concrete call sites instead of replacing the wrapper internals blindly.
8. **Contract gaps**
- Scope: add or repair contracts for the unresolved endpoints listed above, or document why they should remain handwritten.
- This should be separate from service migration PRs.
9. **Special transport cleanup**
- Scope: `ssePost`, `sseGet`, `upload`, public webapp APIs, marketplace APIs, binary/download APIs.
- These should stay last because they rely on special behavior such as stream parsing, abort handling, progress callbacks, webapp passport headers, or marketplace headers.
Suggested PR sequence:
1. `refactor(web): migrate low-risk service APIs to generated contracts`
2. `refactor(web): migrate access control APIs to generated contracts`
3. `refactor(web): migrate knowledge metadata and segment APIs`
4. `refactor(web): migrate dataset service APIs`
5. `refactor(web): migrate app management APIs`
6. `refactor(web): migrate non-streaming workflow APIs`
7. `refactor(web): migrate tools and model provider APIs`
8. `refactor(web): replace dynamic URL API wrappers with contract clients`
9. `chore(contracts): add missing web API contracts for remaining handwritten calls`
Each PR should stay within one domain boundary and avoid mixing contract generation, service migration, page-level behavior changes, and broad test rewrites unless the dependency is unavoidable.
Contributor guide
Research direction
Start with web/service/base.ts, web/service/client.ts, and @dify/contracts, then choose one low-risk static module such as web/service/access-control or billing. Migrate only ordinary JSON calls with directly matched contracts, leaving SSE, uploads, marketplace APIs, and dynamic URL wrappers unchanged. Done means the selected module uses consoleClient or consoleQuery, preserves behavior, and has focused tests where query invalidation or payload shapes are non-trivial.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, frontend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100