chocolatey / chocolatey/choco

[API] No reliable way to query for remote status of multiple packages

Open
#2,116 2 comments 0 reactions 0 assignees View on GitHub
0 - Triaging API Bug Enhancement
Dominant language
C#
Stars
11.5k
Forks
960
PR merge metrics
No merged PRs in 30d

Description

I have a list of packages, *e.g.* `"git vim vlc"`, and I would like to get their respective version information from the online repositories. Ideally this would be through the API, but I realise `choco` doesn’t seem to provide this either.

## Summary of issues

- `choco search` without arguments takes a very long time and downloads a lot of data
- `choco search` without arguments does not actually return all packages
- `--only-id` only works with a single string or substring
- `--page` does not appear to reduce the server load or transferred data amount
- `--id-starts-with` omits a lot of results
- `--page` does not appear to work with `--id-starts-with`

## Description of attempts

Using the command line I can do something like this in bash:

```
for pkg in git vim vlc; do choco search -r -e --by-id-only $pkg; done
```

With the following output:

```
git|2.28.0
vim|8.2.1651
vlc|3.0.11
```

The drawback is that this performs one query per package. If I have 50 packages this becomes cumbersome. With more packages it becomes unrealistic (my IP has been banned from chocolatey.org) and there doesn’t seem to be a middle ground between exact search and “list all packages” that suits my simple need.

I can search for all packages and filter manually using `grep`:

```
choco search -r | grep '^\(git\|vim\|vlc\)|'
```

But this query takes forever (70 seconds) and downloads about 90 MiB of data from the server. Also, plenty of packages are missing (as reported in #2001) and this appears to be non-deterministic.

Chunking the search results in pages (`choco search -r --page-size=100 --page=0`) to reduce the load on the server actually takes exactly as long; it’s as if paging was performed on the client side, making it useless for that purpose.

I noticed that `vlc` and `vim` both start with `v` so I thought I’d search for packages that start with `v` and filter the results, to reduce the work on the server:

```
choco search -r --id-starts-with v | grep '\<\(vim\|vlc\)\>'
````

But while `choco` indeed returns 21 packages starting with `v`, they neither include `vim` nor `vlc`. This may be related to #1836 and the fact that `--id-starts-with` causes `choco` to return significantly fewer packages than usual (in contrast, `choco search -r | grep '^v'` returns 365 packages). So I thought, maybe these packages are on the second result page:

```
choco search -r --id-starts-with v --page 1
```

But this returns zero results. So maybe I can increase the page size:

```
choco search -r --id-starts-with v --page-size 2000
```

But this still returns the same 21 packages as before.

## Feature requests

The help for `choco search` says the argument is a “filter” but does not specify what a filter is. If I understand correctly, it is just a verbatim string (or substring). I think such a syntax extension could be helpful:

```
choco search -r -e --by-id-only git vim vlc # multiple arguments for search
choco search -r -e --by-id-only git,vim,vlc # comma-separated filter
choco search -r -e --by-id-only 'git|vim|vlc' # regular expression
choco search -r -e --by-id-only '(git|vim|vlc)' # alternate regex syntax
```

Note that the Debian package manager allow the following convenient invocation: `apt search --names-only '^(git|vim|vlc)$'`

This may also be what #1345 is about, but the request is a bit unclear to me. There is also [mention of `--order-by-recent`](https://github.com/chocolatey/choco/issues/714#issuecomment-217931062) but I believe this hasn’t been implemented yet.

Contributor guide

Open the contributing guide

Research direction

Start with the choco search entry point and its help text, then trace how --by-id-only, --id-starts-with, --page, and --page-size are handled. Define the intended multi-package query and verify that remote filtering and pagination reduce transferred data while returning all requested package versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.