infiniflow / infiniflow/ragflow

[Feature Request]: I hope to implement a batch parsing queue

Open
#6,540 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

💞 feature
Dominant language
Go
Stars
91k
Forks
10.8k
Avg merge
1d 14h
Merged PRs (30d)
705

Description

Self Checks
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (Language Policy).
  • Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
  • Please do not modify this template :) and fill in all the required fields.
Is your feature request related to a problem?

Describe the feature you'd like

I hope to implement a batch parsing queue with a maximum concurrency of 10. Any tasks exceeding this limit should enter a waiting queue for sequential processing. This is necessary because some APIs restrict concurrent requests, and parsing takes time (e.g., when running overnight). A queuing mechanism is therefore essential.

Describe implementation you've considered

No response

Documentation, adoption, use case

Additional information

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating how parsing tasks are submitted and how concurrent parsing is handled, then determine where a maximum concurrency of 10 and a waiting queue belong. Done should mean excess tasks wait and are processed sequentially without exceeding the concurrency limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.