ClickHouse / ClickHouse/clickhouse-go
DateTime64 server-side parameter values silently truncated to seconds precision
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.3k
- Forks
- 684
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 14
Description
Description
When a time.Time is bound to a server-side parameter typed as DateTime64(N) ({name:DateTime64(6)} syntax) via clickhouse.Named(), the value is formatted with Seconds precision, losing every digit of sub-second precision. The same root cause also wraps the value in toDateTime('<unix>'), which is not a valid DateTime64 literal — so server-side parameter substitution either returns wrong results (when ClickHouse coerces) or fails parameter parsing.
The bug lives in query_parameters.go at the server-side parameter binding entry point:
// query_parameters.go (around L30-L42)
case driver.NamedValue:
if str, ok := p.Value.(string); ok {
options.parameters[p.Name] = str
continue
}
// using the same format logic for NamedValue typed value in function bindNamed
strVal, err := format(timezone, Seconds, p.Value) // <- hardcoded `Seconds`
if err != nil {
return "", err
}
options.parameters[p.Name] = strVal
format(tz, Seconds, t) calls into formatTime(tz, Seconds, value) (bind.go around L235-L273), whose Seconds branch returns toDateTime('<unix>') or toDateTime('YYYY-MM-DD HH:MM:SS') — dropping the sub-second component. This is verified by the existing TestFormatScaledTime (bind_test.go L258-L265):
val, _ := format(t1.Location(), Seconds, t1) // t1 = 2022-01-12 15:00:00.123456789
require.Equal(t, "toDateTime('2022-01-12 15:00:00')", val)
The workaround is clickhouse.DateNamed(name, value, clickhouse.MicroSeconds), which is dispatched into the driver.NamedDateValue branch and formatted via formatTimeWithScale (correct). But callers binding a plain time.Time for a DateTime64(6) column reasonably expect precision to be preserved — they are given silent truncation instead.
Related:
- #615 (closed): introduced
DateNamedas an opt-in workaround for the same root cause on the legacy@namebind path - #1545 (open): same root cause, but for positional
?binding inINSERT ... VALUES (?,?)— sibling surface - Source bug: clickhouse-go #1483 (the parallel report against
clickhouse-connectreferences this defect class)
ClickHouse server version
Code analysis only; not verified against a running server. The local ClickHouse on port 8123 was not reachable during this investigation. The defect is determinable from the formatting logic in bind.go and query_parameters.go and from the existing assertions in bind_test.go.
Reproduction
package issues
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/ClickHouse/clickhouse-go/v2"
clickhouse_tests "github.com/ClickHouse/clickhouse-go/v2/tests"
)
func TestDateTime64NamedParamPrecision(t *testing.T) {
testEnv, err := clickhouse_tests.GetTestEnvironment("issues")
require.NoError(t, err)
conn, err := clickhouse_tests.TestClientWithDefaultSettings(testEnv)
require.NoError(t, err)
t.Cleanup(func() { conn.Close() })
start := time.Date(2021, 1, 1, 13, 37, 42, 1000, time.UTC) // .000001
end := time.Date(2021, 1, 1, 13, 37, 42, 2000, time.UTC) // .000002
const query = `
WITH data AS (
SELECT parseDateTime64BestEffort('2021-01-01 13:37:42.000001', 6) AS t
UNION ALL SELECT parseDateTime64BestEffort('2021-01-01 13:37:42.000002', 6)
UNION ALL SELECT parseDateTime64BestEffort('2021-01-01 13:37:42.000003', 6)
)
SELECT count() FROM data
WHERE t >= {start:DateTime64(6)} AND t <= {end:DateTime64(6)}
`
var n uint64
require.NoError(t, conn.QueryRow(
context.Background(),
query,
clickhouse.Named("start", start),
clickhouse.Named("end", end),
).Scan(&n))
// Expected: 2 (rows .000001 and .000002).
// Actual: 3 — both bindings are truncated to '2021-01-01 13:37:42',
// so the predicate matches every row in that second.
require.Equal(t, uint64(2), n)
}
Substituting clickhouse.DateNamed("start", start, clickhouse.MicroSeconds) (and the same for end) returns the correct count, confirming that the truncation is in the default Named/format(tz, Seconds, …) path — not the wire protocol.
A purely-local demonstration, no server required, against the existing format function:
t1, _ := time.Parse("2006-01-02 15:04:05.000000000", "2021-01-01 13:37:42.000001000")
val, _ := format(t1.Location(), Seconds, t1)
// val == "toDateTime('2021-01-01 13:37:42')" -- microseconds dropped, and
// "toDateTime(...)" is not a valid DateTime64 parameter literal anyway.
Suggested fix (optional)
Two non-fragile options:
- In
query_parameters.go, whenp.Valueis atime.Time(or*time.Time), detect a non-zero sub-second component and emit a literal'YYYY-MM-DD HH:MM:SS.ffffffffff'directly intooptions.parameters[p.Name], bypassingformat()'stoDateTime(...)wrapping. ClickHouse will accept that string for bothDateTime(when no fractional digits are present) andDateTime64(N)parameter types. - Promote
DateNamed: when a caller supplies atime.Timewhose sub-second precision is non-zero, automatically format it at the appropriate scale rather than always atSeconds. Callers with microsecond-zero values are unaffected; callers with sub-second precision get correctness instead of silent truncation. Document the trade-off nearformat()inbind.go.
Either way, the Seconds-by-default behaviour in query_parameters.go:38 is the actual defect — DateNamed should be an explicit-scale convenience, not the mandatory escape hatch for correctness.
Link
Source/related report: https://github.com/ClickHouse/clickhouse-go/issues/1483 (parallel defect class against the clickhouse-connect Python client). Sibling/precursor: #615 (closed), #1545 (open).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in query_parameters.go at the driver.NamedValue binding entry point, then inspect format and formatTime in bind.go. Run the existing TestFormatScaledTime in bind_test.go and add a regression covering plain time.Time with a DateTime64(6) server-side parameter. Done means sub-second precision is preserved without the invalid toDateTime wrapper and the relevant tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100