json_extract() scanning incorrect type for TIMESTAMP GENERATED VIRTUAL columns
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 9.2k
- Forks
- 1.2k
- Avg merge
- 19m
- Merged PRs (30d)
- 4
Description
Hello
I am using SQLite for logging output from my app. Each log entry looks something like this:
{
"level":"info",
"time":"2020-11-30 15:17:41.86803-08:00",
"caller":"cmd/logging.go:124",
"message":"development: true"
}
And the struct I'm using to later retrieve the values looks like this:
type logs struct {
Timestamp time.Time `db:"time"`
Level string `db:"level"`
LogMsg string `db:"log_msg"`
Caller string `db:"caller"`
}
When I use this "logs_not_working" table def and try to pull these values, it fails with an error "sql: Scan error on column index 0, name "time": unsupported Scan, storing driver.Value type string into type *time.Time"
CREATE TABLE IF NOT EXISTS "logs_not_working" (
"time" TIMESTAMP GENERATED ALWAYS AS (json_extract("original", '$.time')) VIRTUAL,
"level" TEXT GENERATED ALWAYS AS (json_extract("original", '$.level')) VIRTUAL,
"log_msg" TEXT GENERATED ALWAYS AS (json_extract("original", '$.message')) VIRTUAL,
"caller" TEXT GENERATED ALWAYS AS (json_extract("original", '$.caller')) VIRTUAL,
"original" TEXT
)
However, if I change the table def to the following and INSERT INTO "logs_working"("time", "original") VALUES("2020-11-30 15:17:41.86803-08:00", "{JSON msg here}"), I am able to extract the rows correctly:
CREATE TABLE IF NOT EXISTS "logs_working" (
"time" TIMESTAMP NOT NULL,
"level" TEXT GENERATED ALWAYS AS (json_extract("original", '$.level')) VIRTUAL,
"log_msg" TEXT GENERATED ALWAYS AS (json_extract("original", '$.message')) VIRTUAL,
"caller" TEXT GENERATED ALWAYS AS (json_extract("original", '$.caller')) VIRTUAL,
"original" TEXT
)
So it appears the driver is not consulting the type hint on GENERATED VIRTUAL columns and properly scanning them. Is this a bug in the driver or is this the expected behavior? I'd prefer to just write the JSON record and let SQLite handle pulling the record apart for me to avoid the overhead of unmarshalling the JSON while logging as well as the extra storage for the double entry for "time"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the generated-column schema and scan into time.Time described in the issue, then compare it with the ordinary TIMESTAMP column behavior. Trace how SQLite column types are exposed to database/sql scanning, and verify whether generated VIRTUAL columns can preserve the declared type; done means the behavior is fixed or clearly documented as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100