hashicorp / hashicorp/nomad

Need a way of starting event stream at the end, or one past last received event

Open
#11,296 3 comments 3 reactions 0 assignees View on GitHub
stage/needs-discussion theme/events type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When making an event subscription, you can optionally pass an index of the start point in the event stream.
The way this seems to work though doesn't seem as useful as it could be.

The behaviour seems to be that if there isn't an event with the requested index, you get the last event. Well, kind of. You'll get that event if it satisfies your subscription filter, otherwise you get nothing until new events are generated that do satisfy your filter.

The trouble with this behaviour seems two fold. There's no apparent way of just starting an event subscription and only receiving new events. The best you can do seems to be to pass in MaxInt as your starting index. This gets rounded down to the index of the last event that exists, and then you either get that event immediately if it matches your filter, or you get nothing until new events arrive that do.
The other problem seems to be that if your connection breaks and you remake the subscription, and you try and track the index of the last event you receive so as to pick up where you left off, you'll almost always get the event twice. So if you try and set the subscription index to "lastEventIndex+1" it says "no event of that index, closest is lastEventIndex, send that one".

Perhaps we're doing this wrong, perhaps I'm reading the code wrong, but this is my experience of what happens, and my reading of what the subscription code does.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the event-subscription behavior described in the issue, then inspect the subscription code and how it handles a requested index beyond the last event. Done means subscriptions can begin with only new events and reconnecting from lastEventIndex+1 does not replay the last event; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.