[Feature] Push jobs should fail eagerly to all data centers if primary data center fails
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 611
- Forks
- 124
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 26
Description
### Feature Request Proposal
#### Background
In Venice's batch push architecture, data is first transmitted to a primary site called the 'nativeReplicationSourceFabric'. Once it's sent there, other sites can download the data.
#### The Ask
If the source fabric isn't able to successfully receive the data in the push for any reason, then there isn't much hope for the other sites to actually finish the job. They should eagerly abort and error the push should the upstream source fail for any reason.
### Motivation
> #### What is the use case for this feature?
This is a fast fail feature meant to save time for users. Currently the job has to time out, and this wastes everyones time, not to mention using some (albeit small) resources on the server.
### Details
_No response_
### What component(s) does this bug affect?
- [X] `Controller`: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
- [ ] `Router`: This is the stateless query-routing layer for serving read requests.
- [ ] `Server`: This is the component that persists all the store data.
- [X] `VenicePushJob`: This is the component that pushes derived data from Hadoop to Venice backend.
- [ ] `VenicePulsarSink`: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.
- [ ] `Thin Client`: This is a stateless client users use to query Venice Router for reading store data.
- [ ] `Fast Client`: This is a stateful client users use to query Venice Server for reading store data.
- [ ] `Da Vinci Client`: This is an embedded, stateful client that materializes store data locally.
- [ ] `Samza`: This is the library users use to make nearline updates to store data.
- [ ] `Admin Tool`: This is the stand-alone client used for ad-hoc operations on Venice.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing failure handling in the VenicePushJob and Controller components, focusing on how a nativeReplicationSourceFabric failure is reported to other data centers. The work is done when a source-fabric receive failure causes the push to abort and error promptly across the other sites instead of waiting for a timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100