langgenius / langgenius/dify

[Refactor/Chore] Migrate handwritten web API helpers to generated contracts

Open
#38,351 2 comments 1 reaction 0 assignees View on GitHub
project#dify refactor web
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

## Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for refactors or chores; if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

## Description

The web frontend still has broad handwritten REST usage through `web/service/base.ts`, even though production API contracts are now available via `@dify/contracts` and `web/service/client.ts` (`consoleClient` / `consoleQuery`).

A local scan of production TypeScript/TSX files found:

- `460` calls to `web/service/base.ts` exported helpers across `62` files.
- `411` ordinary JSON REST calls (`get`, `post`, `put`, `del`, `patch`) across `43` files.
- `47` special calls for SSE, upload, public API, or marketplace API across `23` files.
- `2` utility calls (`request` / `handleStream`).

For the `411` ordinary JSON REST calls, a rough method/path match against the generated console + enterprise contracts found:

- `336` calls directly match generated contracts.
- `66` calls use dynamic URL wrappers and need call-site-level migration.
- `8` calls appear to be missing from the available contracts.
- `1` call partially matches contracts.

High-volume handwritten API files include:

- `web/service/common.ts`: 44 ordinary JSON calls.
- `web/service/datasets.ts`: 32 ordinary JSON calls.
- `web/service/use-plugins.ts`: 27 ordinary JSON calls.
- `web/service/apps.ts`: 25 ordinary JSON calls.
- `web/service/use-pipeline.ts`: 21 ordinary JSON calls.
- `web/service/tools.ts`, `web/service/use-common.ts`, `web/service/use-tools.ts`: 20 ordinary JSON calls each.

Suggested migration scope:

1. Migrate direct, non-streaming console JSON endpoints from `get` / `post` / `put` / `del` / `patch` to `consoleClient` / `consoleQuery`.
2. Start with fully matched, non-dynamic service modules such as access control, knowledge, billing, `use-pipeline`, and `use-tools`.
3. Treat dynamic URL wrappers separately. Examples include generic helpers in `web/service/common.ts`, `web/service/apps.ts`, `web/service/log.ts`, and `web/service/use-plugins-auth.ts`; these should be migrated from the concrete call sites instead of blindly replacing the wrapper internals.
4. Keep `ssePost`, `sseGet`, `upload`, public API helpers, and marketplace helpers for now unless their runtime behavior is explicitly supported by generated clients.

Contract gaps or unresolved paths from the scan:

- `POST /setup`
- `POST /init`
- `GET /init`
- `GET /setup`
- `GET /version`
- `GET /workspaces/current/tool-provider/model/tools`
- `GET /workspaces/current/tool-provider/builtin/{provider}/credentials_schema`
- `GET /auth/plugin/datasource/{pluginId}/{provider}`

Partial contract coverage:

- `GET /apps/{app_id}/workflows/draft/conversation-variables` exists.
- `GET /snippets/{snippet_id}/workflows/draft/conversation-variables` exists.
- `GET /rag/pipelines/{pipeline_id}/workflows/draft/conversation-variables` did not match in the generated contract scan.

## Motivation

The `get`, `post`, `put`, `del`, and `patch` helpers in `web/service/base.ts` are already marked as deprecated for console JSON APIs. Migrating contract-covered endpoints to generated clients should improve:

- Request and response type safety.
- Endpoint drift detection.
- TanStack Query key consistency.
- Reuse of generated contract input/output types.
- Long-term maintainability by reducing handwritten REST helpers.

This should not be a blanket replacement. Streaming, upload, binary, public webapp, and marketplace flows currently rely on behavior in `web/service/base.ts` such as SSE parsing, abort handling, progress handling, webapp passport headers, and marketplace-specific headers. The existing `packages/contracts/non-json-openapi-responses.md` also documents special streaming and binary response behavior that should be considered before migrating those paths.

## Additional Context

Suggested first phase:

- Migrate the `336` directly matched ordinary JSON call sites.
- Leave dynamic URL wrappers and contract gaps as follow-up work.
- Add focused tests around migrated service hooks where behavior, query invalidation, or payload shapes are non-trivial.

Suggested follow-up phase:

- Add or repair contracts for the unmatched endpoints listed above, or explicitly document why they should remain handwritten.
- Refactor dynamic URL wrapper APIs so concrete endpoints are expressed at the call sites and can use generated clients safely.

## Proposed PR Breakdown

This migration should be split by review boundary and runtime risk rather than by raw file count.

Recommended PR groups:

1. **Low-risk direct contract migrations**
- Scope: static endpoints, ordinary JSON responses, no dynamic URL argument, no SSE, no upload.
- Good first migration PR to establish the pattern.
- Candidate modules: `billing`, `use-education`, `strategy`, `use-workspace`, and small fully matched access-control helpers.

2. **Access control / RBAC**
- Scope: `web/service/access-control/*`.
- Reason: clear domain boundary and the codebase already uses some `consoleClient` / `consoleQuery` in this area.

3. **Knowledge / dataset APIs**
- Suggested split:
- Dataset list/detail/settings: `datasets.ts`, `knowledge/use-dataset.ts`.
- Documents, segments, metadata: `knowledge/use-document.ts`, `knowledge/use-segment.ts`, `knowledge/use-metadata.ts`.
- Create/import/hit-testing: `knowledge/use-create-dataset.ts`, `knowledge/use-import.ts`, `knowledge/use-hit-testing.ts`.

4. **Apps / app configuration**
- Scope: `apps.ts`, but split by feature when needed.
- Suggested boundaries: CRUD/copy/import/export, site config/API keys, tracing/webhook.
- Reason: app service changes touch user-facing flows and should stay reviewable.

5. **Workflow non-streaming JSON APIs**
- Scope: non-SSE endpoints in `workflow.ts` and `use-workflow.ts`.
- Explicitly exclude `ssePost`, `sseGet`, `handleStream`, workflow run streams, and resume streams.

6. **Plugins / tools / model provider APIs**
- Suggested split:
- Tool and tool-provider APIs.
- Plugin install/task/list APIs.
- Model-provider APIs.
- Plugin auth dynamic URL APIs as a later PR.
- Keep marketplace APIs out of the first console-contract migration PRs.

7. **Dynamic URL wrappers**
- Scope: wrappers that accept `url`, `urlPath`, or `params.url`, such as parts of `common.ts`, `apps.ts`, `log.ts`, `use-plugins-auth.ts`, and workflow helpers.
- Approach: migrate from concrete call sites instead of replacing the wrapper internals blindly.

8. **Contract gaps**
- Scope: add or repair contracts for the unresolved endpoints listed above, or document why they should remain handwritten.
- This should be separate from service migration PRs.

9. **Special transport cleanup**
- Scope: `ssePost`, `sseGet`, `upload`, public webapp APIs, marketplace APIs, binary/download APIs.
- These should stay last because they rely on special behavior such as stream parsing, abort handling, progress callbacks, webapp passport headers, or marketplace headers.

Suggested PR sequence:

1. `refactor(web): migrate low-risk service APIs to generated contracts`
2. `refactor(web): migrate access control APIs to generated contracts`
3. `refactor(web): migrate knowledge metadata and segment APIs`
4. `refactor(web): migrate dataset service APIs`
5. `refactor(web): migrate app management APIs`
6. `refactor(web): migrate non-streaming workflow APIs`
7. `refactor(web): migrate tools and model provider APIs`
8. `refactor(web): replace dynamic URL API wrappers with contract clients`
9. `chore(contracts): add missing web API contracts for remaining handwritten calls`

Each PR should stay within one domain boundary and avoid mixing contract generation, service migration, page-level behavior changes, and broad test rewrites unless the dependency is unavoidable.

Contributor guide

Open the contributing guide

Research direction

Start with web/service/base.ts, web/service/client.ts, and @dify/contracts, then choose one low-risk static module such as web/service/access-control or billing. Migrate only ordinary JSON calls with directly matched contracts, leaving SSE, uploads, marketplace APIs, and dynamic URL wrappers unchanged. Done means the selected module uses consoleClient or consoleQuery, preserves behavior, and has focused tests where query invalidation or payload shapes are non-trivial.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, frontend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.