ansible / ansible/awx

The project update is triggered every time, when the job template changes the source control branch

Open
#14,432 2 comments 6 reactions 0 assignees View on GitHub
community component:api component:ui type:bug
Dominant language
Python
Stars
15.6k
Forks
3.7k
Avg merge
2d 15h
Merged PRs (30d)
25

Description

### Please confirm the following

- [X] I agree to follow this project's [code of conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html).
- [X] I have checked the [current issues](https://github.com/ansible/awx/issues) for duplicates.
- [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
- [X] I am **NOT** reporting a (potential) security vulnerability. (These should be emailed to `security@ansible.com` instead.)

### Bug Summary

### Context

We were struggling with this bug for a couple of weeks in our RND environment:
https://github.com/ansible/awx/issues/14395
The bug troubleshooting took much more time than expected because of the following bugs in UI (I've opened them accordingly):
https://github.com/ansible/awx/issues/14431
https://github.com/ansible/awx/issues/14430

### The root cause of the issue

We found a root cause of this issue and see that it is related to the project update mechanism that does not work as we expected.

We have many projects and a lot of workflows that are running simultaneously. We defined the **Source Control Branch/Tag/Commit** and **Cache Timeout** for each project and expect the project will not be updated too often because the whole git repo is cached.

But it doesn't work like that. **Each time some job triggers the project with another branch (the branch different from the branch it was triggered before), the project update is triggered**, ignoring the cache. Because of the UI bugs from above, it took us time to find these "invisible" project update jobs.

I'll illustrate this issue with an example. Let's say we have a project named **awx-project**; its **Source Control Branch/Tag/Commit** is set to **master**. We have three workflows: awx-workflow-A (**branch A)**, awx-workflow-B (**branch B**), and awx-workflow-C (**branch C**), that are based on this project. The **project cache** is configured to 20 minutes, so we expect the project update won't be triggered more often than once in 20 minutes. We can even see in UI that it works as expected (and the project update is "kind of" triggered only every 20 minutes), but it does not since there are some hidden project updates we don't see in UI, and we can find them only in logs. Looks like it works for real like that:

1. awx-workflow-A is triggered. The project update has started. The A branch was cached.
2. awx-workflow-A is triggered. The project update is not triggered since branch A is already cached
3. awx-workflow-B is triggered. The project update has started. The B branch was cached.
4. awx-workflow-B is triggered. The project update is not triggered since branch B is already cached
5. awx-workflow-C is triggered. The project update has started. The C branch was cached.
6. awx-workflow-C is triggered. The project update is not triggered since branch C is already cached, and so on...

That means we can trigger only the workflows from the project's default branch; otherwise, it will trigger additional project updates.

So, cache works on the branch level, not the project level. We have many running workflows from the same project but with different branches. It makes the cache configuration almost useless for us. It overloads our system and RDS, so our AWX stops functioning or does it too slowly because all our jobs are waiting for project updates (see the source issue above for all the details).

### AWX version

22.5

### Select the relevant components

- [X] UI
- [ ] UI (tech preview)
- [X] API
- [ ] Docs
- [ ] Collection
- [X] CLI
- [ ] Other

### Installation method

kubernetes

### Modifications

no

### Ansible version

_No response_

### Operating system

_No response_

### Web browser

_No response_

### Steps to reproduce

Run workflows from the same project with different branches.

### Expected results

The whole project is cached according to the Cache configuration.

### Actual results

Only a specific branch is cached, and the cache is overridden for each branch. It makes the project updates triggered too often.

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the behavior with multiple workflows from the same project, each using a different source-control branch, while observing project-update jobs and logs. Trace the project update and cache handling from the API/task-engine entry points; done means the configured cache timeout applies at the project level and different branches do not trigger unnecessary updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.