avro processor decode errors: ErrInputTooLarge and friends reach operators as opaque cerrors, no stable conduiterr.Code
- Dominant language
- Go
- Stars
- 610
- Forks
- 63
- Avg merge
- 12h 28m
- Merged PRs (30d)
- 57
Description
## Summary
`conduit-commons/schema/avro.Serde.Unmarshal` returns typed, stable sentinel errors
(`ErrInputTooLarge`, `ErrSchemaValueMismatch`, `ErrUnsupportedType` — see
`conduit-commons/schema/avro/errors.go`) for known, actionable failure conditions. The engine's
built-in Avro processor throws that signal away.
`pkg/plugin/processor/builtin/impl/avro/internal/decoder.go:53` (`Decoder.Decode`):
```go
err = sch.Unmarshal(data, &out)
if err != nil {
return nil, cerrors.Errorf("failed to unmarshal data with schema (ID: %v): %w", id, err)
}
```
`cerrors.Errorf` with `%w` preserves the underlying error for `errors.Is`/`errors.As`
programmatically, but the operator-facing surface (pipeline status, logs, any UI/CLI that renders
this error) sees only the generic wrapped string. There is no stable `conduiterr.Code` attached, so
an operator whose pipeline is DLQing records because they exceeded a configured
`WithMaxInputSize` ceiling (see `ConduitIO/conduit-commons#278`) gets the same undifferentiated
failure shape as a malformed schema ID, a registry lookup failure, or a genuine decode error —
nothing tells them "this record was rejected for being oversized, here's the configured limit."
This violates `CLAUDE.md`'s "errors are API" principle: "every user-facing error gets a stable
error code, the failing config path, and a suggested fix." `ErrInputTooLarge` already carries
almost everything needed (a stable sentinel, and the message includes the actual size and the
configured limit) — it's just not surfaced as a `conduiterr.Code` at this boundary, the way
`pkg/foundation/cerrors/conduiterr` is designed for.
## Where this was found
Found during an adversarial review of `ConduitIO/conduit-commons#278` (the near-term mitigation
for hamba/avro's archived-codec decoder advisories, `ConduitIO/conduit#2817`) and the paired design
doc (`ConduitIO/conduit#2823`). Out of scope for both — #278 is `conduit-commons`-only and does not
touch this repo; this issue is the correctly-scoped place to fix it.
## Proposed fix (not attempting in this issue)
- Register a `conduiterr.Code` for the avro processor's decode-boundary errors (at minimum,
distinguish "input too large" from "schema/value mismatch" from "unsupported type" from
"registry/header failure"), following the pattern already established elsewhere in
`pkg/foundation/cerrors/conduiterr` (see its package doc for the `New`/`Wrap`/`WithCode`
construction API).
- In `Decoder.Decode`, branch on `errors.Is(err, avro.ErrInputTooLarge)` (and the other
`conduit-commons/schema/avro` sentinels) and wrap with the appropriate `conduiterr.Code` instead
of a bare `cerrors.Errorf`.
- Add a regression test asserting the processor surfaces the correct `conduiterr.Code` (and its
`Reason()`) for an oversized-input rejection, not just that an error occurs.
## Risk tier
Tier 2 (processor, not data-path semantics — this is error-surface plumbing, not a change to
what gets decoded or how).
Contributor guide
Research direction
Read CLAUDE.md and the conduiterr package documentation for the New/Wrap/WithCode API, then inspect pkg/plugin/processor/builtin/impl/avro/internal/decoder.go:53 and conduit-commons/schema/avro/errors.go. Add the decode-boundary error mapping and a regression test for oversized input; done means the processor exposes the appropriate conduiterr.Code and Reason().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100