shard not found exception in corner case
Open
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 18
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
Given the following sequence of events:
- stream is resharded to add more capacity
- worker is unable to finish reading on a shard that was just closed. it dies before marking it as complete.
- next worker comes up, picks up a new shard since the lease on the one that was closed is still valid
- next worker does just fine for more than 24h. Our original shard that was closed has disappeared, but there is still a lease available for that shard in the KCL table
- new workers will try to pick up this lease and immediately die since our shard doesn't exist in our stream, and never have the chance to mark as complete.
This can be fixed with minimal manual intervention in the dynamo table, but maybe attempting to mark a shard as complete when this occurs would prevent processes from churning?
/cc @mick @drboyer
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the worker and KCL lease-handling path against the event sequence described in the issue. Reproduce the missing-shard case if possible; done should mean workers no longer churn when a leased shard has disappeared and the stale lease can be marked complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100