`make proto` generates incompatible protobufs with buf v0.41.0
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v1.1.0-dev
### Issue
Using the wrong version of buf to compile protobufs causes an agent panic.
As of Nomad v1.1.0-dev we need to stick to buf ~v0.30.1~ v0.36.0 which is now installed by `make bootstrap`.
### Reproduction steps
1. Install buf v0.41.0 and run make proto
2. Try to use an external task driver plugin
#### Expected Result
It works. Our wire protocol should always be forward/backward compatible.
#### Actual Result
The Nomad agent panics with:
```
panic: protobuf tag not enough fields in Timestamp.state:
goroutine 350 [running]:
github.com/golang/protobuf/proto.(*unmarshalInfo).computeUnmarshalInfo(0xc000142640)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/table_unmarshal.go:332 +0x1777
github.com/golang/protobuf/proto.(*unmarshalInfo).unmarshal(0xc000142640, 0xc0009c6280, 0xc000850154, 0xc, 0x12, 0x0, 0x2c003b8)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/table_unmarshal.go:136 +0xf45
github.com/golang/protobuf/proto.makeUnmarshalMessagePtr.func1(0xc000850154, 0x12, 0x13, 0xc0001279b0, 0x2, 0xc000507300, 0x2, 0xc000a90320, 0x198, 0x8)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1646 +0x12b
github.com/golang/protobuf/proto.(*unmarshalInfo).unmarshal(0xc000142500, 0xc0001279a0, 0xc000850152, 0x14, 0x14, 0x0, 0x2c003b8)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/table_unmarshal.go:173 +0x88e
github.com/golang/protobuf/proto.makeUnmarshalMessagePtr.func1(0xc000850152, 0x14, 0x15, 0xc0006a36b0, 0x2, 0xc00007e9c0, 0x756ea17d84c242, 0x8013eed091f202a3, 0x0, 0xc00007e888)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1646 +0x12b
github.com/golang/protobuf/proto.(*unmarshalInfo).unmarshal(0xc0001423c0, 0xc0006a36b0, 0xc000850150, 0x16, 0x16, 0x40f36c, 0x3c05390)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/table_unmarshal.go:173 +0x88e
github.com/golang/protobuf/proto.(*InternalMessageInfo).Unmarshal(0x3bd0240, 0x2bc0dc8, 0xc0006a36b0, 0xc000850150, 0x16, 0x16, 0x1, 0x7ff3f8284bb8)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/table_unmarshal.go:63 +0x66
github.com/hashicorp/nomad/plugins/drivers/proto.(*TaskStatsResponse).XXX_Unmarshal(0xc0006a36b0, 0xc000850150, 0x16, 0x16, 0xc0006a36b0, 0x2705101)
go/src/github.com/hashicorp/nomad/plugins/drivers/proto/driver.pb.go:1139 +0x65
github.com/golang/protobuf/proto.(*Buffer).Unmarshal(0xc000626f38, 0x2bc0dc8, 0xc0006a36b0, 0x0, 0x0)
go/src/github.com/hashicorp/nomad/vendor/github.com/golang/protobuf/proto/decode.go:399 +0xa3
google.golang.org/grpc/encoding/proto.codec.Unmarshal(0xc000850150, 0x16, 0x16, 0x25a6fe0, 0xc0006a36b0, 0x2, 0x2)
go/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/encoding/proto/proto.go:93 +0x130
google.golang.org/grpc.recv(0xc000710fc0, 0x7ff3f8378d90, 0x3c05458, 0xc0004b5b00, 0x0, 0x0, 0x25a6fe0, 0xc0006a36b0, 0x7fffffff, 0x0, ...)
go/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/rpc_util.go:711 +0x118
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0004b3880, 0x25a6fe0, 0xc0006a36b0, 0x0, 0xc011f76016e2d671, 0x35d3bec09)
go/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:885 +0xee
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0004b3880, 0x8, 0x2823a2a)
go/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:736 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc0000d0fc0, 0xc0004d9e80, 0xc0004d9e50, 0x30, 0x28)
go/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:594 +0x9f
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0000d0fc0, 0x25a6fe0, 0xc0006a36b0, 0xc00035ab08, 0xc0008cff7a)
go/src/github.com/hashicorp/nomad/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/hashicorp/nomad/plugins/drivers/proto.(*driverTaskStatsClient).Recv(0xc000632e10, 0xc0008cff20, 0x4df569, 0xc0005eaa08)
go/src/github.com/hashicorp/nomad/plugins/drivers/proto/driver.pb.go:4048 +0x62
github.com/hashicorp/nomad/plugins/drivers.(*driverPluginClient).handleStats(0xc00049e840, 0x2bcdf78, 0xc0008785c0, 0xc000403560, 0x2be6e58, 0xc000632e10)
go/src/github.com/hashicorp/nomad/plugins/drivers/client.go:292 +0x7d
created by github.com/hashicorp/nomad/plugins/drivers.(*driverPluginClient).TaskStats
go/src/github.com/hashicorp/nomad/plugins/drivers/client.go:284 +0x3c5
```
Generating protos with buf v0.41.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the make proto and make bootstrap targets, then inspect the generated plugins/drivers/proto/driver.pb.go and the driver client stack shown in the panic. Verify generation with the supported buf version and external task driver use; done means the agent no longer panics and the wire protocol remains compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100