carvel-dev / carvel-dev/ytt

Add a -o jsonpath option and `jsonpath` library module.

Open
#672 2 comments 1 reaction 0 assignees View on GitHub
discussion enhancement
Dominant language
Go
Stars
1.9k
Forks
167
PR merge metrics
No merged PRs in 30d

Description

**Describe the problem/challenge you have**

The usual use case for `ytt` is to generate a set of YAML documents. If working in the Kubernetes space this would often be fed into `kubectl` or `kapp`.

During development and testing one sometimes wants to be able to extract a sub set of resource objects from the generated YAML documents, or you may want to generate summary details on what was generated. Eg., only pass through any Kubernetes `Secret` resources, or show me a list of the resource types, names and namespaces they would be created in.

One can do this by piping the output of `ytt` into a separate tool such as `yq`, or into `jq` if using the `-o json` option, but it would be convenient if there was an inbuilt `-o jsonpath` option which behaved like the same option in `kubectl`. This would especially be of benefit to Kubernetes developers and admins who are familiar with the `kubectl -o jsonpath` behaviour, and they wouldn't need to reach for a different tool like `yq` or `jq` for many tasks.

**Describe the solution you'd like**

Make use of the Kubernetes go-client implementation for `jsonpath` as document in:

* https://pkg.go.dev/k8s.io/client-go/util/jsonpath

Now when we talk about `kubectl` there are two cases to consider. The first is that you query a single Kubernetes resource object. In this case you get back just that resource on which the `jsonpath` query is applied.

When you don't specify a resource by name and give just the type, you will get back a pseudo resource type `List` of form:

```
{
"apiVersion": "v1",
"items": [],
"kind": "List",
"metadata": {
"resourceVersion": ""
}
}
```

This will be used even if there was only one actual resource in the result.

It is proposed that for `ytt` the `-o jsonpath` option should always return a dictionary which follows this form. This could just be:

```
{
"items": []
}
```

but other boilerplate where have `apiVersion` and `List` could be retained, but have the `apiVersion` be a Carvel specific one, like there are pseudo resource tools used in `imgpkg` and `kapp` where there are not equivalents in Kubernetes itself.

By returning a dictionary object all the time, even when there is only one resource, there is consistency and users know that if they know there is only one object they use `.items[0]`, otherwise they have to process all items in the list. If somone has to have special checks for single object vs multiple objects it would be too confusing and hard to write the `jsonpath` queries/transformations.

At the same time as adding the `-o jsonpath` option I would expect to also see a `jsonpath` module added to `ytt` that would make available the same `jsonpath` library support for use in Starlark code of YAML templates, Starlark code files etc.

The only thing I don't know though is whether the `jsonpath` implementation used by the `go-client` is actually a full implementation of `jsonpath` or is a cut down version.

The hope is to have a more complete implementation of a `jsonpath` library such as can see described in:

* https://pypi.org/project/jsonpath-ng/

which implements:

* https://goessner.net/articles/JsonPath/

including arithmetic and binary comparison operators.

It doesn't need to have a separate parse step that library has as that could be dealt with internally, but the idea is that it would return Starlark objects for literals, list, dictionaries etc as would be appropriate for the expression being evaluated. So not just concatenated string result like `kubectl -o jsonpath`.

In effect what would expect to be able to do is:

```
objects = library.get(data.values.resource.type).with_data_values(data.values, plain=True).eval()
result = jsonpath.eval(objects, ".items[*].metadata.name")
```

which would return a list object type in this case.

It is actually a `jsonpath` library module rather than `-o jsonpath` that I am really after as can see some benefits of being able to use `jsonpath` syntax to perform queries and transformations against a set of objects obtained from YAML/JSON decoding of `data.read()`, or the result of executing `eval()` on a library object. One could do some things much more easily using `jsonpath` than hand crafted Starlark code, it could also perhaps be useful direct in YAML templates to construct something on the fly from the `data.values`. I just don't know whether the `go-client` implementation could be used for this purpose as a Starlark module or not, with it being able to do everything the original JSONPath spec defines, with results being native objects for the type of the result.

If I therefore had to choose, I would prefer a full featured `jsonpath` library module over `-o jsonpath`. I thought by also suggesting `-o jsonpath` it might be seen as a more viable option given that it would align better with what is possible with `kubectl`, so thus killing two birds with one stone.

**Anything else you would like to add:**

If a `jsonpath` library module is possible, it would need to treat a YAML document set like a list, a YAML fragment and a struct as a dictionary etc. In other words, it should be pretty transparent and not require converting stuff to native objects first.

_(Also discussed in the [#carvel](https://kubernetes.slack.com/archives/CH8KCCKA5/p1652590916376849) Slack channel)_

---
Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.