bcgov / bcgov/CE_integratedroads

object store connection failures

Open
#45 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
6
Forks
0
PR merge metrics
No merged PRs in 30d

Description

When running `integrate` workflow on the 89 tiles (in parallel), connection to nrs.objectstore.bc.ca can fail on about 10% of the tiles.
Failure is to every connection attempt from a given runner. Rerunning the workflow (failed jobs only) will generally complete.

Issue is not with ogr or with a delay to network initialization for the given runner. The runner cannot connect via curl or after a delay - the runner's ip seems to be blocked/restricted by the object store.

Presumably this is because of the volume of concurrent requests - 89 parallel jobs plus ogr's multithreading requests for parquet files.

There isn't much need for such extreme parallelization - whether the job completes in 10min vs 2hr is unimportant, hopefully restricting the concurrent runners fixes the issue.

Contributor guide

Open the contributing guide

Research direction

Start by locating the `integrate` workflow and its parallel job configuration. Run the workflow or inspect its failed jobs with reduced concurrency, then confirm that all 89 tiles complete without object-store connection failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
cloud, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.