bcgov / bcgov/CE_integratedroads
object store connection failures
- Dominant language
- Shell
- Stars
- 6
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
When running `integrate` workflow on the 89 tiles (in parallel), connection to nrs.objectstore.bc.ca can fail on about 10% of the tiles.
Failure is to every connection attempt from a given runner. Rerunning the workflow (failed jobs only) will generally complete.
Issue is not with ogr or with a delay to network initialization for the given runner. The runner cannot connect via curl or after a delay - the runner's ip seems to be blocked/restricted by the object store.
Presumably this is because of the volume of concurrent requests - 89 parallel jobs plus ogr's multithreading requests for parquet files.
There isn't much need for such extreme parallelization - whether the job completes in 10min vs 2hr is unimportant, hopefully restricting the concurrent runners fixes the issue.
Contributor guide
Research direction
Start by locating the `integrate` workflow and its parallel job configuration. Run the workflow or inspect its failed jobs with reduced concurrency, then confirm that all 89 tiles complete without object-store connection failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100