Designing spec with Kafka streams

https://clojurians.slack.com/archives/C1B1BB2Q3/p1589307392351000

A Kafka Streams topology where conformance is applied automatically to sanitise the input, then split the stream into two - those where conformance resulted in ::s/invalid and a happy path.

In the happy path, a pure function transforms the conformed message. This transformer has a function spec attached to it, but because I have already conformed the input according to the arg spec before calling the function, it will fail the conformance check.

I seem to be facing two options - either remove the function spec, or create yet another spec for the already-conformed input, a "derived spec" of sorts. Neither really feels satisfactory.

Another alternative would be to simply validate with s/valid? earlier in the stream, then conform inside the pure function - which feels like the right thing to do, but it also feels like I am validating the data twice effectively. What would be the cleanest solution here?

The approach we've taken at work is to treat that initial Spec layer as a boundary and assume it maps from "external data format" to "internal data format" and is applied consistently. Thus everything "inside" is Spec'd in terms of the (already conformed) data produced at the boundary.

So if I understand it correctly you have a second spec layer for the internal representation, that represents things that have been conformed, correct?

Yeah, an API Spec, a Domain Spec, and actually a Persistence Spec. The API Spec is conforming, from external data (often strings) to internal data (numeric, Boolean, date, etc). The Domain Spec describes the data formats used inside the API. The Persistence Spec describes the data that goes into the database (flat hash maps with JDBC-compatible types). We don't use all three in all cases, but it seems to have become a useful way to split things up -- and it clearly identifies two boundaries (API input and database output).

That's excellent, thanks. I have used a similar approach in another service, the only thing I'm struggling with in this one is the fact that the internal layer is very thin and quite close to the input layer, thus it feels like duplicating the specs, but might be worth it for the greater clarity. (edited)

spec-coerce is quite good for this too

I would caution against leaning too heavily on conforming with Spec -- I don't really like what spec-coerce enables.

By default it will do the right thing to coerce x to your spec expeceted values, and the other way around (serializing to db type, ex keyword -> string) just requires to override those cases

No spec duplication that way
No polluted specs either (no conform abuse) (edited)

I have used it for some fairly tricky transformations in the past, but I've found it really works well for emulating pattern matching along with multimethods by tagging individual cases through an s/or. In this particular example I only wanted to trim whitespace but for some reason am having trouble fitting it neatly in without overhauling one part of the service or another.

This particular situation also left me wondering whether conformance should either be the final station in your transformation/validation stack or whether it should yield something that also has a spec for it (without further conformers), like @seancorfield described.

In the grand scheme of things I could just trim the fields I need to but with spec around that just feels like the wrong approach.

There are folks on both sides of that decision

Some feel that input should be cleaned and parsed first, then checked for validity with Spec (and therefore next to no conforming). Others feel that input conformance to valid data is a reasonable use of Spec.

I used to have dual specs like you mention, but it requires more work and is more brittle imho

I am slightly on the conforming side of center. spec-coerce is quite a lot further out on the conforming side. Cognitect are on the non-conforming side of center.

spec-coerce builds a separate registry of coercions that leverages specs, which I think is a good approach

It's gotten better lately, it was missing multi-specs, merge & tuple until recently but now it's quite feature complete

It's also 300ish lines of code, easy to modify, fit to your taste if needed

The issue discusses a Kafka Streams topology, Clojure Spec, and spec-coerce, but names no repository files, tests, or entry points. First determine whether an implementation or documentation change is intended; the issue does not currently define a concrete completion condition.