devopshq / devopshq/artifactory-cleanup
`DeleteEmptyFolders` fails on large instances: `.include("*", "property", "stat")` explodes the AQL response (JSONDecodeError)
- Dominant language
- Python
- Stars
- 160
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
**Version:** artifactory-cleanup 1.0.18
### What happens
`DeleteEmptyFolders` issues an unscoped, whole-instance AQL query that the base
policy decorates with `.include("*", "property", "stat")`
(`artifactory_cleanup/rules/base.py:256`):
```
items.find({"$and": [{"path": {"$match": "**"}, "type": {"$eq": "any"}}]}).include("*", "property", "stat")
```
The `property` (and `stat`) include is an AQL **domain join**: Artifactory
returns *one result row per property per artifact*. On a large instance this
inflates the response into millions of rows. `get_artifacts()` then calls
`r.json()` (`base.py:276`) on the full body and dies:
```
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 6587458 column 2 (char 244265313)
File ".../artifactory_cleanup/rules/base.py", line 276, in get_artifacts
content = r.json()
requests.exceptions.JSONDecodeError: ...
```
(~244 MB / 6.5M rows here.) The job had run fine for months; it broke purely
because the instance grew past the point where the response is parseable.
Nothing in our config changed.
### Why it's a bug
Empty-folder detection (`rules/utils.py` → `build_repositories` /
`get_empty_folders`) only reads `repo`, `path`, `name`, and `type`. It never
touches properties or stats. Requesting them is pure overhead — and the
`property` join is precisely what makes the response size unbounded.
### Suggested fix
For `DeleteEmptyFolders`, drop the property/stat joins — `.include("*")` already
returns repo/path/name/type as a single row per artifact:
```python
def aql_add_text(self, aql):
return aql.replace('.include("*", "property", "stat")', '.include("*")')
```
This keeps the all-repositories behavior and removes the row explosion. (More
generally, the per-rule includes could be made minimal rather than always
requesting `"*", "property", "stat"`.)
### Workaround for others hitting this
Subclass the rule and override `aql_add_text` as above, register it, and point
the policy at the subclass.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in artifactory_cleanup/rules/base.py at the AQL decoration around lines 256 and 276, then inspect rules/utils.py functions build_repositories and get_empty_folders. Verify which fields empty-folder detection reads and confirm that DeleteEmptyFolders no longer requests property or stat data while retaining all-repositories behavior; the relevant response should remain parseable without duplicated property rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100