langgenius / langgenius/dify

Improve error handling resilience across extractors and parsers

Open
#39,944 0 comments 2 reactions 0 assignees View on GitHub
🐞 bug
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

1.16.1

### Cloud or Self Hosted

Self Hosted (Docker), Self Hosted (Source)

### Steps to reproduce

This is a preventative maintenance issue discovered via proactive code review. There are no specific end-user steps to reproduce a crash, but the oversights can be observed directly in the codebase:

1. `api/core/rag/extractor/notion_extractor.py`: The `_get_notion_block_data` and `_read_block` methods use an unbounded `while True:` loop for pagination without a safety limit.
2. `api/core/moderation/output_moderation.py`: The `except Exception:` block logs an error but doesn't clarify that the design intentionally fails open.
3. `api/core/agent/output_parser/cot_output_parser.py`: A bare `except:` is used to catch JSON parsing errors, which also catches system signals.
4. `api/core/tools/tool_manager.py`: Tool credential parsing failures are swallowed without any logging.

### ✔️ Expected Behavior

1. Extractors should have sensible pagination limits (e.g., max_pages = 1000) to prevent infinite loops (OOM / CPU lockup).
2. Moderation failure logs should clearly indicate the intentional fail-open behavior to operators.
3. Parsers should only catch specific application exceptions (e.g. json.JSONDecodeError, ValueError).
4. Silent credential fallbacks should emit a warning log for operator visibility.

### ❌ Actual Behavior

1. Unbounded `while True` loop can run infinitely if the Notion API returns unexpected cursor states.
2. Log message is vague (`"Moderation Output error"`) and could cause operators to think the app crashed.
3. Bare `except:` catches `KeyboardInterrupt` and `SystemExit`, preventing graceful shutdowns.
4. Credential parsing silently falls back to `{}` with no log, making debugging broken tool auth impossible.

Contributor guide

Open the contributing guide

Research direction

Start by reading the four named entry points: api/core/rag/extractor/notion_extractor.py, api/core/moderation/output_moderation.py, api/core/agent/output_parser/cot_output_parser.py, and api/core/tools/tool_manager.py. Trace the pagination, moderation exception, JSON parsing, and credential fallback paths. Done means bounded pagination, explicit fail-open logging, specific parsing exceptions, and warning visibility for credential failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.