yesodweb / yesodweb/persistent
SQLite backend silently loses precision for Ratio and Fixed
Nobody has claimed this yet.
- Dominant language
- Haskell
- Stars
- 486
- Forks
- 306
- PR merge metrics
- No merged PRs in 30d
Description
SQLite doesn't support arbitrary precision and silently converts things to floats if they're too large to fit in 64 bits. From the documentation:
INTEGER. The value is a signed integer, stored in 1, 2, 3, 4, 6, or 8 bytes depending on the magnitude of the value.
REAL. The value is a floating point value, stored as an 8-byte IEEE floating point number.
[...]
A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT. For the purposes of this paragraph, hexadecimal integer literals are not considered well-formed and are stored as TEXT. (This is done for historical compatibility with versions of SQLite prior to version 3.8.6 2014-08-15 where hexadecimal integer literals were first introduced into SQLite.) No attempt is made to convert NULL or BLOB values.
The following examples demonstrates the problem:
module Main where
import Control.Monad.Trans
import Data.Ratio
import Database.Persist.Sqlite
import Database.Persist.TH
share [mkPersist sqlSettings, mkMigrate "migrateAll"] [persistLowerCase|
TestEntity
val Rational
deriving Show
|]
main :: IO ()
main = do
runSqlite ":memory:" $ do
runMigration migrateAll
entId <- insert $ TestEntity (10^100 % 1)
entRead <- get entId
liftIO $ print entRead
It prints:
Migrating: CREATE TABLE "test_entity"("id" INTEGER PRIMARY KEY,"val" NUMERIC(32,20) NOT NULL)
Just (TestEntity {testEntityVal = 10000000000000000159028911097599180468360808563945281389781327557747838772170381060813469985856815104 % 1})
We could represent Ratios, Fixeds, and Integers as text values, setting the column type affinity to TEXT in migrations. But that would mean the comparison operators return the wrong results in some cases, since it converts the operands to numbers beforehand rather than doing strcmp. There's no good answers afaict.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Database.Persist.Sqlite backend and reproduce the in-memory example using Data.Ratio and the generated TestEntity. Investigate how Ratio, Fixed, and Integer values are stored and compared in SQLite, then define an approach that preserves precision without breaking comparison behavior; done requires a resolved design and regression coverage for large values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell, sqlite
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100