jet / jet/equinox

Cosmos: Support competing writers with inconsistent `unfold`s

Open
#108 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
F#
Stars
501
Forks
70
PR merge metrics
No merged PRs in 30d

Description

At present, if you supply an `unfold` function, for each write, its output:
a) travels on the wire to the stored procedure (this does not cost extra, so optimizing that is not really a concern)
b) replaces the current `u`nfolds in the `Tip` document (as usual, you pay for a read with the original size and a write with the new size (not forgetting the cost of the read per active ChangeFeedProcessor) in terms of Request Charges)

Pros and cons:
+ if you stop writing an unfold, it will get removed on the next write, saving storage space, and RUs on the write, subsequent reads, and induced reads by ChangeFeedProcessor(s)
- if you are doing blue/green deploys, old and new writers can interleave - the loser needs to rebuild their state unless the competing writer happens to also have written the same one (and then potentially remove the competitor's one iff it writes)

A small change to the JS can improve the semantics by
a) removing unfolds that are being updated and replacing with the supplied values
b) retaining any that are not being touched (perhaps subject to some retention criteria, i.e. only keep it if it's <= N events behind Tip)
c) allowing the writer to indicate a set of `c`aseNames that should be removed regardless of retention rules

Related: there's a `tip-isa-batch` branch which stores events in the Tip - the competing writers scenario's efficiency would be greatly improved by this (any conflict will yield the competing events cleanly, and any unfolds that are behind will typically see both their unfold and successor events from the single point-read roundtrip)

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by locating the Cosmos JavaScript stored procedure and the Tip document handling for unfolds, then clarify the retention and removal semantics for interleaved writers. Done means the agreed competing-writer behavior, including explicit caseNames removal, is implemented and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, fsharp, javascript
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.