tonsky / tonsky/datascript

Must throw exception if eid's are too big

Open
#56 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Clojure
Stars
5.8k
Forks
318
PR merge metrics
No merged PRs in 30d

Description

I was experimenting with external ids by using the second approach suggested by @tonsky at https://github.com/tonsky/datascript/issues/31#issuecomment-57993865 and found an undocumented max entity-id limit exists at 0x1FFFFFFF but surpassing it gets unnoticed. Check the following code and his output:

(doseq [base-id [800 0x1FFFFFFF (inc 0x1FFFFFFF) 0xFFFFFFFFFFFFFFFF]]
  (let [db                (data/create-conn {})
        {:keys [tempids]} (data/transact! db [{:db/id base-id
                                               :name "Foo"}
                                              {:db/id -1  
                                              :name "Bar"}
                                              {:db/id -2
                                               :name "Baz"}
                                              ])]
    (println "Using base-id = " base-id)
    (println "  =>  tempids = " tempids)
    (println "  =>  entity  = " (-> @db (data/entity base-id) data/touch))
    (println "  => :max-eid = " (:max-eid @db))
    (println)
    ))

Output:

 Using base-id =  800
   =>  tempids =  {-1 801, -2 802, :db/current-tx 536870913}
   =>  entity  =  {:name Foo, :db/id 800}
   => :max-eid =  802

 Using base-id =  536870911
   =>  tempids =  {-1 536870912, -2 536870912, :db/current-tx 536870913}
   =>  entity  =  {:name Foo, :db/id 536870911}
   => :max-eid =  536870911

 Using base-id =  536870912
   =>  tempids =  {-1 1, -2 2, :db/current-tx 536870913}
   =>  entity  =  {:name Foo, :db/id 536870912}
   => :max-eid =  2

 Using base-id =  18446744073709552000
   =>  tempids =  {-1 1, -2 2, :db/current-tx 536870913}
   =>  entity  =  {:name Foo, :db/id 18446744073709552000}
   => :max-eid =  2

Notes and questions:

1- When using base-id = 536870911 (0x1FFFFFFF) datascript can't assign new temporal ids for the next entities, they all get 536870912 (0x1FFFFFFF + 1). I think an exception must be raised.

2- When using base-id >= (0x1FFFFFFF + 1) the database :max-eid is not updated, but you can sucesfully retrieve the created entity with the big id. Is ok and safe to use big ids greater than 0x1FFFFFFF + 1?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied transact! reproduction with the listed base-id values and inspect how :max-eid and tempids are handled. Trace the data/transact! and data/entity entry points to determine the expected behavior for ids beyond 0x1FFFFFFF. Done means oversized ids no longer silently produce invalid temporal ids, and the supported behavior for larger ids is documented or enforced.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.