tonsky / tonsky/datascript

creating entity IDs from squuids

Open
#168 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Clojure
Stars
5.8k
Forks
318
PR merge metrics
No merged PRs in 30d

Description

Hi, I'm trying to figure out how to keep entity IDs created locally by datascript in a central database.

The IDs in the central database need to be more universally unique, to allow more than one client access to the data at a time.

I've written a bit of code to generate a squuid for each entity ID as it is sent to the remote server and make sure that squuids and entity IDs are matched consistently across multiple transactions/datoms.

I'm now not sure what the best way to get squuids back into entity IDs on a fresh page load (or injected into an existing database) would be.

I have a few ideas but I'm not totally happy with any of them:

  • Use SQUUIDS as entity IDs in datascript -> currently unsupported, throws an error because datascript expects integer entity IDs
  • Increment a counter for each new squuid hitting the browser from the database, use the counter as the entity ID -> there is no guarantee that squuids will be sent to the browser in the order they were created, making the order of the entity IDs different to the order of the squuids (which is bad if you rely on the order of entity IDs)
  • Use the time component of the SQUUID directly as the entity ID -> this results in entity IDs that are larger than the start of transaction IDs, leading to at least one pretty bad bug in advance-max-eid where the database's max-eid is not updated. Even if we subtract a big number from the time before using it, we can still easily get collisions as the time component is only accurate to the second

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named. Start by tracing advance-max-eid and the existing entity-ID handling, then review how SQUUIDs and entity IDs are matched across transactions and restored on page load. Done requires an agreed approach that works for fresh and existing databases without the listed ordering, collision, or max-eid problems.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.