mattn / mattn/go-sqlite3

Seserialize and Deserialize failed for the size greater than 2G

Open
#1,309 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
9.2k
Forks
1.2k
Avg merge
19m
Merged PRs (30d)
4

Description

When the size of db exceeds 2G (approximately the size ofint32), the call of Serialize (also Deserialize) will fail.

Here is a minimal poc to reproduce the problem

package main

import (
	"context"
	"database/sql"
	"fmt"
	"os"
	"strings"

	"github.com/mattn/go-sqlite3"
)

func must(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	db, err := sql.Open("sqlite3", ":memory:")
	must(err)

	_, err = db.Exec("create table test (id integer primary key, name text)")
	must(err)

	b := strings.Builder{}
	for i := 0; i < 10000; i++ {
		b.WriteString("a")
	}

	rows := 200000
	// rows := 300000 // changed rows to 300000 cause serialize to fail
	for i := 0; i < rows; i++ {
		_, err = db.Exec("insert into test (name) values (?)", b.String())
		must(err)
	}

	// serialize to file
	conn, err := db.Conn(context.Background())
	must(err)

	err = conn.Raw(func(driverConn any) error {
		sqliteConn := driverConn.(*sqlite3.SQLiteConn)
		bs, err := sqliteConn.Serialize("")
		if err != nil {
			return err
		}
		f, err := os.Create("poc.db")
		if err != nil {
			return err
		}
		_, err = f.Write(bs)
		return err
	})
	must(err)

	// deserialize from file
	db, err = sql.Open("sqlite3", ":memory:")
	must(err)
	conn, err = db.Conn(context.Background())
	must(err)
	err = conn.Raw(func(driverConn any) error {
		sqliteConn := driverConn.(*sqlite3.SQLiteConn)
		bs, err := os.ReadFile("poc.db")
		if err != nil {
			return err
		}
		return sqliteConn.Deserialize(bs, "")
	})
	must(err)
	conn.Close()

	r, err := db.Query("select count(*) from test")
	must(err)
	defer r.Close()
	var count int
	r.Next()
	must(r.Scan(&count))
	fmt.Println(conut)

}

I did not look too deep of the issues, but I suspected the problem may related to the conversion between uint64 and uintptr. I noticed the sqlite3_malloc64 in https://github.com/mattn/go-sqlite3/blob/7658c06970ecf5588d8cd930ed1f2de7223f1010/sqlite3_opt_serialize.go#L69 returns null when the length of b is greater than 2G.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sqlite3_opt_serialize.go around line 69 and reproduce the failure using the minimal Go PoC, comparing database sizes below and above 2G. Trace the Serialize and Deserialize paths and the reported sqlite3_malloc64 allocation result. Done means databases larger than 2G can be serialized and deserialized successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.