buildkite / buildkite/agent

Support directories in `artifact_paths`

Open
#1,570 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

Suppose I want to upload all contents of a directory `foo/bar`. I would love to specify artifact paths like this:

```yaml
artifact_paths:
- foo/bar
```

I would expect this to be treated the same as `foo/bar/*`. But actually no artifacts are uploaded, even if the directory exists and is non-empty. This is further complicated by the name of the variable/key. The problem is that`artifact_paths` are not paths, they are _patterns_. It is even more confusing that they do not support directories which _are_ paths!

Suppose I have a command step that wants to create one directory from `$BUILDKITE_ARTIFACT_PATHS` if it doesn't exist. How would I go about it without first-class directory support?

Well, let's try the common shell script and assume we have the following glob pattern configured:

```yaml
artifact_paths:
- foo/**/*
```

In our command:

```ShellSession
# Try to create directory foo
mkdir $BUILDKITE_ARTIFACTS_PATHS
```

... oops, now we created a directory called `foo/**/*`, what I really wanted was to create a directory called `foo` and put all my artifacts in that directory.

This is even _more_ confusing as a user because of the existence of `nullglob`, `dotglob` and the other variety of shell options. Maybe we should all stop using shell scripts (I don't disagree). But I think for now, it would be better to support directories in `artifact_paths`.

**Describe the solution you'd like**
If a path `foo` is specified in `artifact_paths` and it is a directory that exists, it should be treated like `foo/*` during the upload step.

Edit: I think the directory should be treated like `foo/**/*`

Contributor guide

Open the contributing guide

Research direction

Start by locating the Go code that parses artifact_paths and performs artifact uploads, then find its related tests and existing glob-pattern handling. Verify how an existing directory is detected and define completion as uploading its contents recursively, matching the requested foo/**/* behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ci-cd, devops
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.