Improve error handling resilience across extractors and parsers
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
1.16.1
### Cloud or Self Hosted
Self Hosted (Docker), Self Hosted (Source)
### Steps to reproduce
This is a preventative maintenance issue discovered via proactive code review. There are no specific end-user steps to reproduce a crash, but the oversights can be observed directly in the codebase:
1. `api/core/rag/extractor/notion_extractor.py`: The `_get_notion_block_data` and `_read_block` methods use an unbounded `while True:` loop for pagination without a safety limit.
2. `api/core/moderation/output_moderation.py`: The `except Exception:` block logs an error but doesn't clarify that the design intentionally fails open.
3. `api/core/agent/output_parser/cot_output_parser.py`: A bare `except:` is used to catch JSON parsing errors, which also catches system signals.
4. `api/core/tools/tool_manager.py`: Tool credential parsing failures are swallowed without any logging.
### ✔️ Expected Behavior
1. Extractors should have sensible pagination limits (e.g., max_pages = 1000) to prevent infinite loops (OOM / CPU lockup).
2. Moderation failure logs should clearly indicate the intentional fail-open behavior to operators.
3. Parsers should only catch specific application exceptions (e.g. json.JSONDecodeError, ValueError).
4. Silent credential fallbacks should emit a warning log for operator visibility.
### ❌ Actual Behavior
1. Unbounded `while True` loop can run infinitely if the Notion API returns unexpected cursor states.
2. Log message is vague (`"Moderation Output error"`) and could cause operators to think the app crashed.
3. Bare `except:` catches `KeyboardInterrupt` and `SystemExit`, preventing graceful shutdowns.
4. Credential parsing silently falls back to `{}` with no log, making debugging broken tool auth impossible.
Contributor guide
Research direction
Start by reading the four named entry points: api/core/rag/extractor/notion_extractor.py, api/core/moderation/output_moderation.py, api/core/agent/output_parser/cot_output_parser.py, and api/core/tools/tool_manager.py. Trace the pagination, moderation exception, JSON parsing, and credential fallback paths. Done means bounded pagination, explicit fail-open logging, specific parsing exceptions, and warning visibility for credential failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100