cockroachdb / cockroachdb/cockroach
s390x: node cannot join a cluster created by amd64 nodes, vectorized engine panics deserializing Arrow batches
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Describe the problem
A CockroachDB v26.2.5 node running on s390x (IBM Z, big endian) cannot start when joining a cluster that was created by amd64 nodes. It registers in the cluster, then panics during server controller startup and exits. The node never becomes live, and it crash loops.
The same binary works fine in two other configurations, so this is specific to mixing architectures:
| Cluster created by | Node joining | Result |
|---|---|---|
| s390x, single node | n/a | works |
| s390x, multi node | s390x | works |
| amd64 | s390x | **crashes** |
The failure is in the DistSQL network receive path, deserializing an Apache Arrow batch sent by a remote node.
## Error
```
ERROR: server startup failed: cockroach server exited with error: failed to start the server controller: list-tenants: unexpected error from the vectorized engine: runtime error: slice bounds out of range [:100663296] with capacity 8
SQLSTATE: XX000
```
## Environment
s390x node:
```
$ uname -m
s390x
$ lscpu | grep Endian
Byte Order: Big Endian
$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.4 (Plow)"
$ cat /proc/cpuinfo | head
vendor_id : IBM/S390
# processors : 4
bogomips per cpu: 24038.00
max thread id : 0
features : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt sie
```
amd64 nodes:
```
$ uname -m
x86_64
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
$ cat /proc/cpuinfo | head
vendor_id : AuthenticAMD
model name : AMD EPYC 7B12
```
Version on all three nodes: `cockroachdb/cockroach:v26.2.5`, `CCL v26.2.5 @ 2026/07/28 18:55:44 (go1.25.5)`.
The two amd64 nodes run in GCP. The s390x node is a LinuxONE instance. They are connected over a WireGuard overlay (10.99.0.0/24), MTU 1380, round trip latency about 50ms. Clocks are NTP synced on both sides.
## Steps to reproduce
1. Start two amd64 nodes and run `cockroach init` from one of them, so the cluster is created by amd64.
2. Start an s390x node with `--join` pointing at the amd64 nodes.
3. The s390x node registers, then panics and exits.
```
docker run -d --name roach-z --network host --restart unless-stopped \
-v roachz:/cockroach/cockroach-data \
cockroachdb/cockroach:v26.2.5 start --insecure \
--listen-addr=10.99.0.2:26257 --advertise-addr=10.99.0.2:26257 \
--http-addr=0.0.0.0:8080 \
--locality=region=ibm-z,arch=s390x \
--join=10.99.0.1:26257,10.99.0.1:26357,10.99.0.2:26257
```
## Observed: node registers with correct metadata, then dies
```
id | address | sql_address | build | started_at | updated_at | locality | attrs | is_available | is_live
-----+-----------------+-----------------+---------+--------------------------------------+--------------------------------------+--------------------------+-------+--------------+----------
1 | 10.99.0.1:26257 | 10.99.0.1:26257 | v26.2.5 | 2026-08-01 02:09:01.073841 +0000 UTC | 2026-08-01 02:29:01.101834 +0000 UTC | region=gcp-us,arch=amd64 | [] | true | true
2 | 10.99.0.2:26257 | 10.99.0.2:26257 | v26.2.5 | 2026-08-01 02:28:25.451943 +0000 UTC | 2026-08-01 02:28:26.37248 +0000 UTC | region=ibm-z,arch=s390x | [] | false | false
3 | 10.99.0.1:26357 | 10.99.0.1:26357 | v26.2.5 | 2026-08-01 02:09:02.011637 +0000 UTC | 2026-08-01 02:29:02.027139 +0000 UTC | region=gcp-us,arch=amd64 | [] | true | true
(3 rows)
```
Address, build and locality are all correct. It just cannot stay up. Networking is not the issue.
## Stack trace
Log prefixes stripped for readability. I have the full log if it is useful.
```
Wraps: (2) list-tenants
Wraps: (3) plan gist: AgEQBgAhAgAAExACAwUCEQYC
Wraps: (4) while executing: SELECT _ FROM _ WHERE ((_ = $1) AND (_ = $1)) AND (_ IS NOT NULL) ORDER BY _
Wraps: (5) assertion failure
Wraps: (6) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
| pkg/sql/colexecerror/error.go:77
| runtime.gopanic
| GOROOT/src/runtime/panic.go:783
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.InternalError
| pkg/sql/colexecerror/error.go:301
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next.func1
| pkg/sql/colflow/colrpc/inbox.go:331
| runtime.gopanic
| GOROOT/src/runtime/panic.go:783
| runtime.goPanicSliceAcap
| GOROOT/src/runtime/panic.go:141
| github.com/apache/arrow/go/arrow/array.(*Binary).ValueBytes
| external/com_github_apache_arrow_go_arrow/array/binary.go:83
| github.com/cockroachdb/cockroach/pkg/col/colserde.getValueBytesAndOffsets
| pkg/col/colserde/arrowbatchconverter.go:588
| github.com/cockroachdb/cockroach/pkg/col/colserde.(*ArrowBatchConverter).ArrowToBatch
| pkg/col/colserde/arrowbatchconverter.go:422
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*Deserializer).Deserialize.func1
| pkg/sql/colexec/colexecutils/deserializer.go:69
| github.com/cockroachdb/cockroach/pkg/sql/colmem.(*Allocator).PerformOperation
| pkg/sql/colmem/allocator.go:443
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*Deserializer).Deserialize
| pkg/sql/colexec/colexecutils/deserializer.go:68
| github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Inbox).Next
| pkg/sql/colflow/colrpc/inbox.go:415
Wraps: (7) unexpected error from the vectorized engine
Wraps: (8) runtime error: slice bounds out of range [:100663296] with capacity 8
```
Higher up the chain:
```
github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
pkg/server/server_controller.go:350
github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
pkg/server/server_controller.go:239
github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
pkg/server/server_controller.go:188
github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
pkg/server/server.go:2398
```
## What I ruled out
**Not a packaging or build difference.** The binary inside the image is byte identical to the published s390x tarball:
```
$ docker run --rm --entrypoint sha256sum cockroachdb/cockroach:v26.2.5 /cockroach/cockroach
dc34cc6410bc0a668ae1859ef40784b44e2c5d884e6898ee1dc58d924efb034a /cockroach/cockroach
$ sha256sum ~/cockroach-v26.2.5.linux-s390x/cockroach
dc34cc6410bc0a668ae1859ef40784b44e2c5d884e6898ee1dc58d924efb034a /home/linux1/cockroach-v26.2.5.linux-s390x/cockroach
```
**Not "s390x is broken".** Single node works:
```
$ ./cockroach-v26.2.5.linux-s390x/cockroach start-single-node --insecure --listen-addr=127.0.0.1:26999 --http-addr=127.0.0.1:8099
CockroachDB node starting at 2026-08-01 03:12:35.607071869 +0000 UTC m=+0.789653496 (took 0.5s)
build: CCL v26.2.5 @ 2026/07/28 18:55:44 (go1.25.5)
status: initialized new cluster
nodeID: 1
```
**Not the join path by itself.** Two s390x nodes forming their own cluster work fine:
```
$ docker exec -it z1 cockroach node status --insecure
id | address | sql_address | build | started_at | updated_at | locality | attrs | is_available | is_live
-----+----------+-------------+---------+--------------------------------------+--------------------------------------+--------------------+-------+--------------+----------
1 | z1:26257 | z1:26257 | v26.2.5 | 2026-08-01 03:32:30.449005 +0000 UTC | 2026-08-01 03:32:48.466333 +0000 UTC | arch=s390x,node=z1 | [] | true | true
2 | z2:26257 | z2:26257 | v26.2.5 | 2026-08-01 03:32:31.411348 +0000 UTC | 2026-08-01 03:32:49.417531 +0000 UTC | arch=s390x,node=z2 | [] | true | true
(2 rows)
```
The all-s390x case ran as two containers on one host over a Docker bridge, and the amd64 pair was co-located in GCP, so the failing case also differs in that the crashing node is remote across the WireGuard link. I have not isolated locality from architecture with a same-endianness pair across the same link. That said, the crash is deterministic and reproduces on every restart of the mixed cluster, and the panic value below points at endianness rather than a dropped or truncated packet.
## Observation on the panic value
`100663296` is `0x06000000`, which is `6` with the bytes reversed. In the failing query, `list-tenants` runs `SELECT name FROM system.tenants`, a query over a STRING column. The panic is a slice bound of `100663296` against a real capacity of `8`, in `Inbox.Next` while deserializing an Arrow batch received from a remote node. A byte-swapped small integer being used as a length is consistent with a byte-order mismatch in the received batch, though I have not traced the serialization code to confirm the exact field.
For what it is worth on the code side: a GitHub code search for `BigEndian` in `pkg/col/colserde` returns nothing, and `system.BigEndian` (defined in `pkg/util/system/endian.go`) appears only in two test files (`pkg/sql/logictest/logic.go` and `pkg/sql/colflow/routers_test.go`). I have not run this down in a local checkout, so treat that as a search result rather than a definitive claim.
## Workaround
Setting the cluster-wide vectorize default to off lets the s390x node start and join:
```
SET CLUSTER SETTING sql.defaults.vectorize = 'off';
```
After that, the same mixed cluster is healthy:
```
id | address | sql_address | build | started_at | updated_at | locality | attrs | is_available | is_live
-----+-----------------+-----------------+---------+--------------------------------------+--------------------------------------+--------------------------+-------+--------------+----------
1 | 10.99.0.1:26257 | 10.99.0.1:26257 | v26.2.5 | 2026-08-01 02:09:01.073841 +0000 UTC | 2026-08-01 02:38:25.10025 +0000 UTC | region=gcp-us,arch=amd64 | [] | true | true
2 | 10.99.0.2:26257 | 10.99.0.2:26257 | v26.2.5 | 2026-08-01 02:37:40.138038 +0000 UTC | 2026-08-01 02:38:25.177011 +0000 UTC | region=ibm-z,arch=s390x | [] | true | true
3 | 10.99.0.1:26357 | 10.99.0.1:26357 | v26.2.5 | 2026-08-01 02:09:02.011637 +0000 UTC | 2026-08-01 02:38:26.027412 +0000 UTC | region=gcp-us,arch=amd64 | [] | true | true
(3 rows)
```
The setting turns off vectorized execution cluster-wide, which is a performance cost, so it is a stopgap rather than a fix.
## Impact
Mixed amd64 and s390x clusters cannot be formed on v26.2.5 with default settings. Single architecture clusters of either kind (all amd64 or all s390x) are unaffected and run on defaults. This matters for hybrid deployments that place CockroachDB on IBM Z alongside x86.
Contributor guide
Research direction
Reproduce the mixed amd64/s390x cluster failure with vectorized execution enabled, then read pkg/col/colserde/arrowbatchconverter.go around ArrowToBatch and getValueBytesAndOffsets, followed by pkg/sql/colflow/colrpc/inbox.go and deserializer.go. Check the endian references in pkg/util/system/endian.go and the existing colflow router tests. Done means an s390x node can join an amd64-created cluster with vectorization enabled without the deserialization panic, with regression coverage for the failing path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100