apache / apache/accumulo

Modify FATE to allow distributed workers

Open
#3,822 2 comments 0 reactions 1 assignee Claimed by @keith-turner View on GitHub
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

FATE could be modified to allow distributed workers to run FATE transactions. Currently FATE relies on the manager lock to ensure one FATE instance is running. That single instance uses a in memory synchronized java map to reserve FATE transaction so that only a single thread runs. Could move to a model of using a condition update (in ZK or metadata table if #3559 is implemented) to reserve a FATE transaction, setting the process lock id per transaction. Would need to have a process of deleting locks held by dead processes.

Distributing FATE could allow scalability of operations like bulk import. It may also be an alternative to using a distributed work queue for the problem of split computation #3560 and compaction file selection #3526. This may be a prerequisite for #3247

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.