clojure.spec practices - validating data

So far I've been using spec mostly for validating data structures "in-place", meaning I have an x, at the front door of my API I check if it's a valid format, and if it is I continue using x inside my API. There's another way of using spec that I don't really understand yet so I'm looking for good resources on it: I have an x, and at the front door of my API I check if it's a valid format and destructure/reformat it to the format expected internally in my API, y. y might have all the same data as x but perhaps arranged in a different nested structure or with different key names. Are there any good guides to using spec for this kind of thing? I guess this is using spec as a parser, rather than just a validator. This use case is hinted at (https://clojure.org/about/spec#_conform) but I haven't found a good set of examples yet.

alexmiller
That’s what conform is for
“Is this valid, and if so, why?”
With the caveat that it is NOT for “perform arbitrary transformation to other structure”

Ben Grabow 17:27
I'm trying to understand the situations it's good for. e.g. https://juxt.pro/blog/posts/parsing-with-clojure-spec.html
alexmiller 17:27
If you want that, use conform + standard Clojure data transformation
Ben Grabow 17:27
In that example, a sequence of characters is being parsed into a map that describes the components.
17:27
I have one map with a nested structure that I want to transform to another map with a flat structure.
alexmiller 17:27
Yeah, dont do that
17:28
(That applies to the string case)
17:28
If you want to transform maps, use Clojure stuff to do that
17:28
If you want to validate, use valid?
17:29
If you want to know why something is valid wrt options and alternatives, use conform
Ben Grabow 17:29
It's that last part I'm looking for more content on.
17:30
I don't quite understand when I'm dealing with a situation that's good for conform and when I'm not.
17:33
I remember in one of Rich's talks he mentioned David Nolen rewriting part of the cljs parser in spec. Is that code publicly available somewhere? It sounds like a powerful use case.
Ben Grabow 17:41
http://blog.cognitect.com/blog/2017/1/3/spec-destructuring
Cognitect BlogCognitect Blog
Creating a spec for destructuring
Cognitect news, musings and more.
5 Jan 2017
alexmiller 17:42
Well that’s mine :)
17:42
Using it to parse the input to a complex macro is a good fit
17:42
It’s particularly good for “syntax” stuff
17:43
If your input is mostly maps and stuff, probably not as useful
Ben Grabow 17:45
The difference in the way my input map is structured vs how my output map is structured feels like syntax to me so I wonder where the distinction lies
17:47
There's the idea that a map is just a seq of kv pairs. What if the spec is an s/cat of kv-pairs instead of an s/keys?
alexmiller 17:49
Syntax is positional (this is kind of present in the Greek roots of the word even)
17:49
Maps are not positional
17:50
Position in syntax is implicit
17:50
Spec regex ops describe that, and the conformed values tell you how it was interpreted
17:51
Must of the map specs essentially return the map if it’s valid - there was no thing to figure out and tell you about
17:53
Specing a map as a seq of kv pairs is not the same thing as maps are inherently unordered
17:53
You can’t rely on position to describe the structure of the map
Ben Grabow 17:54
hmm, that's a good point
alexmiller 17:54
The regex specs like cat will even error if you try to do this, for this reason
Ben Grabow 17:56
I had in mind using a big s/or to describe all the possible keys. Which brings up an interesting question, how does the positionality of syntax impact s/or? I usually reach for s/or when I have a couple different possibilities for my input, like a union type, and I'm usually frustrated that I'm pushed towards destructuring and naming the options instead of conforming to just the value.
17:56
(s/or :even even? :small #(< % 42)) from the docstring as an example.
17:57
I wish the API were (s/or even? #(< % 42)) for the use I have in mind
17:57
I feel like there's something I don't "get" in the reasons why s/or asks for keywords to name the options.
alexmiller 17:59
or gives you back a map entry of tag (key) and value
Ben Grabow 18:02
Sure, I get that. I'm wondering why it does that when it doesn't seem to be about processing the position of a thing. Instead it seems to be about processing the alternatives of a single thing.
alexmiller 18:27
well the idea is the same - tell me what choice you made when there was an alternative
18:28
in practice, it is often a source of issues in deeply nested data where that's the only conformed thing so I kind of would like to have some more options in spec 2
18:28
whether that's an or variant, or a s/nonconforming, or something else
18:29
but that's down the list a bit and we haven't talked about it
Ben Grabow 18:33
Glad to hear the sentiment is shared.
18:35
I often run into situations where a map is mostly just a set of specific things so s/keys works, until we add uuids into the mix and they could be strings or uuid objects. I'm using a custom conformer to get around the s/or destructuring for that case but it would be nice to have a more general solution.
alexmiller 18:39
one workaround right now is to wrap those s/or's in s/nonconforming
18:39
which is not documented, but exists
Ben Grabow 18:49
Nice!
18:49

(comment
(s/conform (s/nonconforming (s/or :x uuid? :x ::sc/uuid-string)) (UUID/randomUUID))
(s/conform (s/nonconforming (s/or :x uuid? :x ::sc/uuid-string)) (str (UUID/randomUUID)))
(gen/generate (s/gen (s/nonconforming (s/or :x uuid? :x ::sc/uuid-string)))))

All these work as expected. The first two produce just the value, without the keyword tag. The third generates both strings and uuids.
18:51
My ::uuid-string spec is still a little complicated to make the generator work but I think it could be simplified.

Start with the clojure.spec conform documentation and the linked parsing-with-clojure-spec and spec-destructuring resources. Determine whether this issue calls for a guide covering conform, s/or, s/nonconforming, and map transformation, since the thread does not identify files or tests. Done would require an agreed documentation scope and corresponding examples.