ClickHouse / ClickHouse/clickhouse-go

DateTime64 server-side parameter values silently truncated to seconds precision

Open
#1,858 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3.3k
Forks
684
Avg merge
2d 3h
Merged PRs (30d)
14

Description

Description

When a time.Time is bound to a server-side parameter typed as DateTime64(N) ({name:DateTime64(6)} syntax) via clickhouse.Named(), the value is formatted with Seconds precision, losing every digit of sub-second precision. The same root cause also wraps the value in toDateTime('<unix>'), which is not a valid DateTime64 literal — so server-side parameter substitution either returns wrong results (when ClickHouse coerces) or fails parameter parsing.

The bug lives in query_parameters.go at the server-side parameter binding entry point:

// query_parameters.go (around L30-L42)
case driver.NamedValue:
    if str, ok := p.Value.(string); ok {
        options.parameters[p.Name] = str
        continue
    }
    // using the same format logic for NamedValue typed value in function bindNamed
    strVal, err := format(timezone, Seconds, p.Value)   // <- hardcoded `Seconds`
    if err != nil {
        return "", err
    }
    options.parameters[p.Name] = strVal

format(tz, Seconds, t) calls into formatTime(tz, Seconds, value) (bind.go around L235-L273), whose Seconds branch returns toDateTime('<unix>') or toDateTime('YYYY-MM-DD HH:MM:SS') — dropping the sub-second component. This is verified by the existing TestFormatScaledTime (bind_test.go L258-L265):

val, _ := format(t1.Location(), Seconds, t1)            // t1 = 2022-01-12 15:00:00.123456789
require.Equal(t, "toDateTime('2022-01-12 15:00:00')", val)

The workaround is clickhouse.DateNamed(name, value, clickhouse.MicroSeconds), which is dispatched into the driver.NamedDateValue branch and formatted via formatTimeWithScale (correct). But callers binding a plain time.Time for a DateTime64(6) column reasonably expect precision to be preserved — they are given silent truncation instead.

Related:

  • #615 (closed): introduced DateNamed as an opt-in workaround for the same root cause on the legacy @name bind path
  • #1545 (open): same root cause, but for positional ? binding in INSERT ... VALUES (?,?) — sibling surface
  • Source bug: clickhouse-go #1483 (the parallel report against clickhouse-connect references this defect class)

ClickHouse server version

Code analysis only; not verified against a running server. The local ClickHouse on port 8123 was not reachable during this investigation. The defect is determinable from the formatting logic in bind.go and query_parameters.go and from the existing assertions in bind_test.go.

Reproduction

package issues

import (
	"context"
	"testing"
	"time"

	"github.com/stretchr/testify/require"

	"github.com/ClickHouse/clickhouse-go/v2"
	clickhouse_tests "github.com/ClickHouse/clickhouse-go/v2/tests"
)

func TestDateTime64NamedParamPrecision(t *testing.T) {
	testEnv, err := clickhouse_tests.GetTestEnvironment("issues")
	require.NoError(t, err)
	conn, err := clickhouse_tests.TestClientWithDefaultSettings(testEnv)
	require.NoError(t, err)
	t.Cleanup(func() { conn.Close() })

	start := time.Date(2021, 1, 1, 13, 37, 42, 1000, time.UTC) // .000001
	end := time.Date(2021, 1, 1, 13, 37, 42, 2000, time.UTC)   // .000002

	const query = `
WITH data AS (
    SELECT parseDateTime64BestEffort('2021-01-01 13:37:42.000001', 6) AS t
    UNION ALL SELECT parseDateTime64BestEffort('2021-01-01 13:37:42.000002', 6)
    UNION ALL SELECT parseDateTime64BestEffort('2021-01-01 13:37:42.000003', 6)
)
SELECT count() FROM data
WHERE t >= {start:DateTime64(6)} AND t <= {end:DateTime64(6)}
`

	var n uint64
	require.NoError(t, conn.QueryRow(
		context.Background(),
		query,
		clickhouse.Named("start", start),
		clickhouse.Named("end", end),
	).Scan(&n))

	// Expected: 2 (rows .000001 and .000002).
	// Actual:   3 — both bindings are truncated to '2021-01-01 13:37:42',
	// so the predicate matches every row in that second.
	require.Equal(t, uint64(2), n)
}

Substituting clickhouse.DateNamed("start", start, clickhouse.MicroSeconds) (and the same for end) returns the correct count, confirming that the truncation is in the default Named/format(tz, Seconds, …) path — not the wire protocol.

A purely-local demonstration, no server required, against the existing format function:

t1, _ := time.Parse("2006-01-02 15:04:05.000000000", "2021-01-01 13:37:42.000001000")
val, _ := format(t1.Location(), Seconds, t1)
// val == "toDateTime('2021-01-01 13:37:42')"  -- microseconds dropped, and
// "toDateTime(...)" is not a valid DateTime64 parameter literal anyway.

Suggested fix (optional)

Two non-fragile options:

  1. In query_parameters.go, when p.Value is a time.Time (or *time.Time), detect a non-zero sub-second component and emit a literal 'YYYY-MM-DD HH:MM:SS.ffffffffff' directly into options.parameters[p.Name], bypassing format()'s toDateTime(...) wrapping. ClickHouse will accept that string for both DateTime (when no fractional digits are present) and DateTime64(N) parameter types.
  2. Promote DateNamed: when a caller supplies a time.Time whose sub-second precision is non-zero, automatically format it at the appropriate scale rather than always at Seconds. Callers with microsecond-zero values are unaffected; callers with sub-second precision get correctness instead of silent truncation. Document the trade-off near format() in bind.go.

Either way, the Seconds-by-default behaviour in query_parameters.go:38 is the actual defect — DateNamed should be an explicit-scale convenience, not the mandatory escape hatch for correctness.

Link

Source/related report: https://github.com/ClickHouse/clickhouse-go/issues/1483 (parallel defect class against the clickhouse-connect Python client). Sibling/precursor: #615 (closed), #1545 (open).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in query_parameters.go at the driver.NamedValue binding entry point, then inspect format and formatTime in bind.go. Run the existing TestFormatScaledTime in bind_test.go and add a regression covering plain time.Time with a DateTime64(6) server-side parameter. Done means sub-second precision is preserved without the invalid toDateTime wrapper and the relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.