[Question] Limitation and strategies for extensive parallelism
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 169
Description
Hi all! I am currently using workflows for extraction, but I expect to receieve extremely large files from around 10,000 chunks. These chunks go straight to a SoE pipeline with 3 calls to an llm provider. What are the limitations of workflows to handle this scenarios? Should I take advantage of the idempotency behavior of steps and create my own queue process?
Would love some feedback or insight from you, awesome work!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the workflow and step execution model, focusing on limits for roughly 10,000 chunks and three LLM calls per chunk. Compare the built-in idempotency behavior with the proposed queue approach, then document the supported constraints and recommended strategy; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100