devopshq / devopshq/artifactory-cleanup

`DeleteEmptyFolders` fails on large instances: `.include("*", "property", "stat")` explodes the AQL response (JSONDecodeError)

Open Beginner friendly
#188 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
160
Forks
87
PR merge metrics
No merged PRs in 30d

Description

**Version:** artifactory-cleanup 1.0.18

### What happens

`DeleteEmptyFolders` issues an unscoped, whole-instance AQL query that the base
policy decorates with `.include("*", "property", "stat")`
(`artifactory_cleanup/rules/base.py:256`):

```
items.find({"$and": [{"path": {"$match": "**"}, "type": {"$eq": "any"}}]}).include("*", "property", "stat")
```

The `property` (and `stat`) include is an AQL **domain join**: Artifactory
returns *one result row per property per artifact*. On a large instance this
inflates the response into millions of rows. `get_artifacts()` then calls
`r.json()` (`base.py:276`) on the full body and dies:

```
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 6587458 column 2 (char 244265313)
File ".../artifactory_cleanup/rules/base.py", line 276, in get_artifacts
content = r.json()
requests.exceptions.JSONDecodeError: ...
```

(~244 MB / 6.5M rows here.) The job had run fine for months; it broke purely
because the instance grew past the point where the response is parseable.
Nothing in our config changed.

### Why it's a bug

Empty-folder detection (`rules/utils.py` → `build_repositories` /
`get_empty_folders`) only reads `repo`, `path`, `name`, and `type`. It never
touches properties or stats. Requesting them is pure overhead — and the
`property` join is precisely what makes the response size unbounded.

### Suggested fix

For `DeleteEmptyFolders`, drop the property/stat joins — `.include("*")` already
returns repo/path/name/type as a single row per artifact:

```python
def aql_add_text(self, aql):
return aql.replace('.include("*", "property", "stat")', '.include("*")')
```

This keeps the all-repositories behavior and removes the row explosion. (More
generally, the per-rule includes could be made minimal rather than always
requesting `"*", "property", "stat"`.)

### Workaround for others hitting this

Subclass the rule and override `aql_add_text` as above, register it, and point
the policy at the subclass.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in artifactory_cleanup/rules/base.py at the AQL decoration around lines 256 and 276, then inspect rules/utils.py functions build_repositories and get_empty_folders. Verify which fields empty-folder detection reads and confirm that DeleteEmptyFolders no longer requests property or stat data while retaining all-repositories behavior; the relevant response should remain parseable without duplicated property rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.