[Performance] Log pages extremely slow on large DB (~740GB): N+1 loading of workflow_runs TOAST fields + content search without usable index
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the Contributing Guide
- [x] This is only for bug report, if you would like to ask a question, please head to Discussions
- [x] I have searched for existing issues, including closed ones
- [x] I confirm that I am using English to submit this report
### Dify version
v1.16.1
### Cloud or Self Hosted
Self Hosted (Docker Compose)
### Problem
The application log pages (workflow logs / chat logs) become extremely slow when the
database grows large, especially when searching logs by conversation content
(keyword search). In our production environment:
- PostgreSQL database is ~740 GB, with `workflow_runs` and `workflow_node_executions`
consuming over 97% of the storage (mostly TOAST columns: `graph`, `inputs`, `outputs`, `error`)
- Peak QPS ~100 (customer-service chatflow)
- 6 API replicas, SQLAlchemy pool size 8 + overflow 3 per replica
### Steps to reproduce
1. Open the app log page (e.g. `/apps/{app_id}/monitor/logs`)
2. Enter a keyword in the search box to filter logs by conversation/message content
3. Observe the request time
### Actual Behavior
- The first page of the log list triggers per-row lookups against `workflow_runs`
(N+1 pattern): each `workflow_app_log` row loads its related `workflow_run` and
deserializes large TOAST fields (`graph`, `inputs`, `outputs`, `error`)
- Keyword search by conversation content scans a huge `messages` table without an
index that can serve the predicate, taking tens of seconds or more
- On the large database the log page often times out (504 behind a reverse proxy)
- The response payload is several MB because graph snapshots are embedded per row
### Expected Behavior
- Log list loads with bounded, predictable latency regardless of total data volume
- Keyword/content search uses a queryable path (proper index, full-text search, or an
external log store) instead of full scans on multi-hundred-GB tables
- Avoid N+1 per-row loading of large TOAST columns for list views
### Additional context
- 1.16.1 still serializes log list rows through `WorkflowAppLog.workflow_run`
(workflow_app_log controller/service path), so even with `detail=false` the per-row
`workflow_runs` lookups are not fully eliminated
- The `workflow_app_logs (tenant_id, app_id, created_at DESC)` index makes the latest
first page fast, but it does not help content search or the per-row run-detail loading
- We validated with EXPLAIN that the page-1 SQL itself is sub-millisecond; the cost is
the row-by-row run-detail loading + TOAST decompression + serialization + large JSON
payload transfer to the browser
- Happy to provide `EXPLAIN ANALYZE` output, slow query logs, or a pg_stat_statements
dump on request
Contributor guide
Assessment
This issue has not been assessed yet.