Unstructured-IO / Unstructured-IO/unstructured-python-client

Problem in using unstructured workflows

Open
#254 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
119
Forks
22
Avg merge
1d 21h
Merged PRs (30d)
1

Description

Hi!
I am developing an ETL pipeline using restate and Unstructured, using python. I have the following problem:

  • I do not want to use Partition Endpoint because I also need the enrichment step.
  • Using a workflow, I noticed that if a second user requests to process a document while the first one is still processing, there are two possible cases:
      1. the second request is quite close to the first (i.e., the first initiated job is still reading files from the source (in my case a folder on S3))
      1. the second request occurs during a later stage of the job: in this case the request is lost (a request consists of loading a new file into the source folder of S3 and then starting the workflow)

Since the first case is also problematic (if multiple users upload files almost simultaneously they will have to wait for all files to be processed before seeing their output, since unstructured first processes and then saves everything together), I came up with this solution:

  • When a new request comes in, I create a source folder on S3, specific to that request.
  • I create and start a workflow specific to that request, which I then destroy when finished
    This way there should be no problem, do you have any better suggestions?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository files or tests are identified. Start by reviewing the workflow entry point and the concurrent-request behavior described for S3-backed processing, then clarify the expected handling of requests arriving during each workflow stage. Done would require an agreed and reproducible resolution for overlapping uploads and workflow runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.