langgenius / langgenius/dify

Add deterministic local and staging E2E capacity benchmarks for Dify Agent

Open
#41,139 0 comments 1 reaction 1 assignee Claimed by @zhaohao1004 View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

## Problem

Dify Agent lacks a reproducible benchmark harness that can separately validate:

- controlled local Runtime throughput;
- the real E2B execution path and lifecycle constraints;
- end-user-visible Staging throughput and replica scaling.

Terminal success alone is not sufficient evidence for Config and File workloads. The benchmark also needs to prove SSE terminal semantics, deterministic results, exact file bytes and SHA256, and physical resource cleanup.

## Proposed scope

- Add deterministic `local-runtime` and `local-e2b` capacity matrices.
- Add the `staging-public` Service API smoke and R1/R2/R4 scaling harness.
- Cover Basic, Shell, Resume, Config, and deterministic 16 MiB File contracts.
- Use Locust closed-loop load with per-user isolation, fixed windows, and terminal drain.
- Validate SSE terminal semantics, markers, Tool/File output, exact bytes, and SHA256.
- Collect E2B running/paused counts locally without a Kubernetes observer Job.
- Reconcile Conversation, Workspace, Binding, ToolFile, storage, and Vendor resources.
- Redact credentials and private identifiers from public artifacts.
- Document single-block/shared-Staging confidence and E2B inventory limitations.

## Acceptance criteria

- Focused tests cover load phases, scenarios, schemas, cleanup, redaction, and reports.
- Local benchmark checks and benchmark-scoped type checking pass.
- File success requires ledger, replay, exact 16 MiB, SHA256, and zero cleanup evidence.
- Staging reports do not claim Agent capacity when the load ceiling is E2B-limited.
- No credential or private resource identifier appears in public artifacts.

## Non-goals

- Production SLOs or maximum production concurrency.
- E2B vendor capacity attribution.
- Cost or linear-scaling guarantees.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.