carvel-dev / carvel-dev/vendir

vendir support for "dot" paths without conflict

Open
#244 4 comments 4 reactions 0 assignees View on GitHub
carvel-accepted enhancement
Dominant language
Go
Stars
399
Forks
73
Avg merge
2h 53m
Merged PRs (30d)
2

Description

**Describe the problem/challenge you have**

vendir's requirement that paths do not overlap makes it difficult, or impossible, to create a directory structure that custom ytt modules can be found in if there are also environment specific yamls which need to be included.

consider the following.

* a _ytt_lib directory needs to be included from a git location
* some number of blobs of yaml (which i'll just call "modules" for now) need to be included
* some number of environment specific yaml one-off files need to be included.

The final directory structure that we're looking to create wants to look like this

```
├── config
   ├── _ytt_lib
   │   └── ambassador-mappings
   │   └── deployment.yaml
   ├── modules
   │   ├── base
   │   │   ├── defaults.yaml
   │   │   ├── deployment.yaml
   │   │   ├── namespace.yaml
   │   │   ├── service-account.yaml
   │   │   └── service.yaml
   │   ├── cdm-subscription
   │   │   └── deployment.yaml
   │   └── ...
   ├── secret.enc.yaml
   ├── terraform.yaml
   └── values.yaml
```

The above directory layout does not seem to be possible due to a constraint in vendir; `Expected to not manage overlapping paths: 'config/modules' and 'config'`

The vendir.yml file we're making using of looks roughly like the following.

```
---
apiVersion: "vendir.k14s.io/v1alpha1"
kind: "Config"
directories:
- path: "config"
contents:
- path: "."
git:
url: "https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git"
ref: "v1.28.1"
includePaths:
- "k8s/envs/stg-01/secret.enc.yaml"
newRootPath: "k8s/envs/stg-01"
- path: "config/service"
contents:
- path: "modules"
git:
url: "https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git"
ref: "v1.28.1"
includePaths:
- "k8s/modules/base/*"
- "k8s/modules/cdm-subscription/*"
- "k8s/modules/cloud-resources/*"
- "k8s/modules/cloud-account-credentials/*"
- "k8s/modules/csp/*"
- "k8s/modules/data-reconciler/*"
- "k8s/modules/flagger-canary/*"
- "k8s/modules/inventory-new-accounts/*"
- "k8s/modules/irsa-service-account-annotation/*"
- "k8s/modules/k8s-collector/*"
- "k8s/modules/kapp-config/*"
- "k8s/modules/legacy-changes/*"
- "k8s/modules/pod-disruption-budget/*"
- "k8s/modules/service-annotations/*"
- "k8s/modules/service-endpoints/*"
- "k8s/modules/wavefront/*"
newRootPath: "k8s/modules"
- path: "config/_ytt_lib"
contents:
- path: "ambassador-mappings"
git:
url: "https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git"
ref: "v1.28.1"
includePaths:
- "k8s/modules/ambassador-mappings/*"
newRootPath: "k8s/modules/ambassador-mappings"
```

the first blob is the blob at issue because it wants to add content to ".", but "." conflicts with the overlay constraint.

**Describe the solution you'd like**

I'd like vendir to allow this use case similar to how `gilt` allows it. For example, the following `gilt` configuration produces the desired result.

```
---
- git: https://gitlab.foo.bar/vss-cloud-accounts-service-infra.git
version: v1.28.1
files:
- src: k8s/modules
dst: config/modules
post_commands:
- rm -rf ambassador-mappings
- src: k8s/modules/ambassador-mappings
dst: config/_ytt_lib/ambassador-mappings
- src: k8s/envs/stg-01/*
dst: config/
post_commands:
- rm -rf vendir.yml
```

but gilt itself has its own constraints in terms of functionality. constraints that vendir doesnt have.

**Anything else you would like to add:**

part of the issue here, as far as I understand things, has to do with the `ytt` requirement that the `_ytt_lib` directory must be a sibling of any yaml files that use content in that directory.

If this `ytt` constraint did not exist, then there would be a clear path forward in how to use vendir. We could create any number of sub-directories and then ytt could either be told where it's `_ytt_lib` is, or, `ytt` could walk up the tree until it found a suitable `_ytt_lib` to use.

---
Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the overlapping-path error with the vendir.yml configuration in the issue, then trace vendir's path-overlap validation. Define and test the intended behavior for a content path of "." alongside nested paths; done means the configuration succeeds and produces the requested config/modules, config/_ytt_lib, and environment-specific files without conflicts.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, go
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.