ConduitIO / ConduitIO/conduit

avro processor decode errors: ErrInputTooLarge and friends reach operators as opaque cerrors, no stable conduiterr.Code

Open
#2,824 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

## Summary

`conduit-commons/schema/avro.Serde.Unmarshal` returns typed, stable sentinel errors
(`ErrInputTooLarge`, `ErrSchemaValueMismatch`, `ErrUnsupportedType` — see
`conduit-commons/schema/avro/errors.go`) for known, actionable failure conditions. The engine's
built-in Avro processor throws that signal away.

`pkg/plugin/processor/builtin/impl/avro/internal/decoder.go:53` (`Decoder.Decode`):

```go
err = sch.Unmarshal(data, &out)
if err != nil {
return nil, cerrors.Errorf("failed to unmarshal data with schema (ID: %v): %w", id, err)
}
```

`cerrors.Errorf` with `%w` preserves the underlying error for `errors.Is`/`errors.As`
programmatically, but the operator-facing surface (pipeline status, logs, any UI/CLI that renders
this error) sees only the generic wrapped string. There is no stable `conduiterr.Code` attached, so
an operator whose pipeline is DLQing records because they exceeded a configured
`WithMaxInputSize` ceiling (see `ConduitIO/conduit-commons#278`) gets the same undifferentiated
failure shape as a malformed schema ID, a registry lookup failure, or a genuine decode error —
nothing tells them "this record was rejected for being oversized, here's the configured limit."

This violates `CLAUDE.md`'s "errors are API" principle: "every user-facing error gets a stable
error code, the failing config path, and a suggested fix." `ErrInputTooLarge` already carries
almost everything needed (a stable sentinel, and the message includes the actual size and the
configured limit) — it's just not surfaced as a `conduiterr.Code` at this boundary, the way
`pkg/foundation/cerrors/conduiterr` is designed for.

## Where this was found

Found during an adversarial review of `ConduitIO/conduit-commons#278` (the near-term mitigation
for hamba/avro's archived-codec decoder advisories, `ConduitIO/conduit#2817`) and the paired design
doc (`ConduitIO/conduit#2823`). Out of scope for both — #278 is `conduit-commons`-only and does not
touch this repo; this issue is the correctly-scoped place to fix it.

## Proposed fix (not attempting in this issue)

- Register a `conduiterr.Code` for the avro processor's decode-boundary errors (at minimum,
distinguish "input too large" from "schema/value mismatch" from "unsupported type" from
"registry/header failure"), following the pattern already established elsewhere in
`pkg/foundation/cerrors/conduiterr` (see its package doc for the `New`/`Wrap`/`WithCode`
construction API).
- In `Decoder.Decode`, branch on `errors.Is(err, avro.ErrInputTooLarge)` (and the other
`conduit-commons/schema/avro` sentinels) and wrap with the appropriate `conduiterr.Code` instead
of a bare `cerrors.Errorf`.
- Add a regression test asserting the processor surfaces the correct `conduiterr.Code` (and its
`Reason()`) for an oversized-input rejection, not just that an error occurs.

## Risk tier

Tier 2 (processor, not data-path semantics — this is error-surface plumbing, not a change to
what gets decoded or how).

Contributor guide

Open the contributing guide

Research direction

Read CLAUDE.md and the conduiterr package documentation for the New/Wrap/WithCode API, then inspect pkg/plugin/processor/builtin/impl/avro/internal/decoder.go:53 and conduit-commons/schema/avro/errors.go. Add the decode-boundary error mapping and a regression test for oversized input; done means the processor exposes the appropriate conduiterr.Code and Reason().

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.