JanusGraph / JanusGraph/janusgraph

LogProcessor is not processing event logs from beginning.

Open
#1,053 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

Starting a LogProcessor to process log fom Begining using setStartTime(Instant.EPOCH) doesn't work.
Following is a sample code used.
I have validated that, the event log is created and persisted in HBase Janusgraph tabe.

JanusGraphTransaction tx = graph.buildTransaction().logIdentifier("addedPerson").start();

JanusGraphVertex u = tx.addVertex("human");
u.property("name", "proteros");
u.property("age", 36);

JanusGraphVertex u1 = tx.addVertex("human");
u.property("name", "sandeep");
u.property("age", 34);
tx.commit();

and trying to replay it using below code.

LogProcessorFramework logProcessor1 = JanusGraphFactory.openTransactionLog(graph);
logProcessor1.addLogProcessor("addedPerson").setStartTime(Instant.EPOCH).setProcessorIdentifier("addedPersonProcessor").setStartTime(Instant.EPOCH).addProcessor(new ChangeProcessor() {
@Override
public void process(JanusGraphTransaction janusGraphTransaction, TransactionId transactionId, ChangeState changeState) {
System.out.println("processing old log");
for (Vertex v : changeState.getVertices(Change.ADDED)) {
System.out.println("vertex " + v.label());
if (v.label().equals("human")) totalHumansAdded.incrementAndGet();
}
}
}).build();

However, my logProcessor doesn't process any previously saved vertices and only detect changes which are performed in after the LogProcessor has been built.

I tried setting up Start Time to INSTANT.EPOCH, which as per documentation should make log processing to start from BEGINING but no help.
The only difference is see in logs is the MessagePuller is started from EPOCH.

22:59:19,039 INFO KCVSLog:744 - Loaded unidentified ReadMarker start time 2018-05-01T14:59:19.023Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@57db2b13
22:59:19,054 INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@2f953efd
22:59:19,054 INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@14bdbc74
22:59:19,054 INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@5a7fe64f
22:59:19,054 INFO KCVSLog:748 - Loaded indentified ReadMarker start time 1970-01-01T02:18:20Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@41330d4f

Contributor guide

Open the contributing guide

Research direction

Start with the LogProcessorFramework and JanusGraphFactory.openTransactionLog entry points, then inspect how setStartTime(Instant.EPOCH) reaches the MessagePuller. Reproduce the example with a persisted event log and compare replay from EPOCH with events created after the processor is built. Done means previously saved vertices are processed as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.