Seserialize and Deserialize failed for the size greater than 2G
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 9.2k
- Forks
- 1.2k
- Avg merge
- 19m
- Merged PRs (30d)
- 4
Description
When the size of db exceeds 2G (approximately the size ofint32), the call of Serialize (also Deserialize) will fail.
Here is a minimal poc to reproduce the problem
package main
import (
"context"
"database/sql"
"fmt"
"os"
"strings"
"github.com/mattn/go-sqlite3"
)
func must(err error) {
if err != nil {
panic(err)
}
}
func main() {
db, err := sql.Open("sqlite3", ":memory:")
must(err)
_, err = db.Exec("create table test (id integer primary key, name text)")
must(err)
b := strings.Builder{}
for i := 0; i < 10000; i++ {
b.WriteString("a")
}
rows := 200000
// rows := 300000 // changed rows to 300000 cause serialize to fail
for i := 0; i < rows; i++ {
_, err = db.Exec("insert into test (name) values (?)", b.String())
must(err)
}
// serialize to file
conn, err := db.Conn(context.Background())
must(err)
err = conn.Raw(func(driverConn any) error {
sqliteConn := driverConn.(*sqlite3.SQLiteConn)
bs, err := sqliteConn.Serialize("")
if err != nil {
return err
}
f, err := os.Create("poc.db")
if err != nil {
return err
}
_, err = f.Write(bs)
return err
})
must(err)
// deserialize from file
db, err = sql.Open("sqlite3", ":memory:")
must(err)
conn, err = db.Conn(context.Background())
must(err)
err = conn.Raw(func(driverConn any) error {
sqliteConn := driverConn.(*sqlite3.SQLiteConn)
bs, err := os.ReadFile("poc.db")
if err != nil {
return err
}
return sqliteConn.Deserialize(bs, "")
})
must(err)
conn.Close()
r, err := db.Query("select count(*) from test")
must(err)
defer r.Close()
var count int
r.Next()
must(r.Scan(&count))
fmt.Println(conut)
}
I did not look too deep of the issues, but I suspected the problem may related to the conversion between uint64 and uintptr. I noticed the sqlite3_malloc64 in https://github.com/mattn/go-sqlite3/blob/7658c06970ecf5588d8cd930ed1f2de7223f1010/sqlite3_opt_serialize.go#L69 returns null when the length of b is greater than 2G.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with sqlite3_opt_serialize.go around line 69 and reproduce the failure using the minimal Go PoC, comparing database sizes below and above 2G. Trace the Serialize and Deserialize paths and the reported sqlite3_malloc64 allocation result. Done means databases larger than 2G can be serialized and deserialized successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100