Reprocessing can cause ingestion of new packages to be delayed by over an hour

If we are running a full reprocessing of all packages on Construct Hub, it's possible that so many fargate tasks will be running that it's impossible for new packages to get processed. (Specifically, the "Orchestration" step function may have a task running to ingest a package, and it may get very unlucky retrying the ECS SubmitTask operation in an exponential backoff loop we have defined).

Fixingn this may require separating the compute for reprocessing and regular ingestion, or adding some way to prioritize new package tasks over reprocessing package tasks, or something else entirely.

Related: https://github.com/cdklabs/construct-hub/issues/708

Start by reviewing the Orchestration step function and its ECS SubmitTask retry behavior, then read related issue #708. Define how regular ingestion should be prioritized or isolated from full reprocessing, and consider the work done only when new packages are no longer delayed by reprocessing load.

Written by the indexing model from the issue text.