AnswerDotAI / AnswerDotAI/ghapi
GhApi.packages breaks when packages have a `/` in them
- Lingua principale
- Python
- Stelle
- 685
- Fork
- 70
- Merge medio
- 1m
- PR unite (30g)
- 2
Descrizione
Note: Since I'm dealing with a private organization with private repositories and private containers, I've changed everything to `myorg`, `myrepo`, and `mycontainer`, respectively.
## Summary
- It appears that Ghapi is using urllib.parse [quote on route items](https://github.com/fastai/ghapi/commit/e3977220b9b158c72c460adda47818847c26beaa).
- But the [default behavior](https://docs.python.org/3/library/urllib.parse.html#url-quoting) of `urllib.parse.quote` treats `/` as a safe character.
- GHCR packages that belong to a particular repository have a `/` in their name and must be referenced by the API with an escaped `/` (i.e. %2F) (e.g. `/orgs/myorg/packages/container/myrepo%2Fmycontainer` is valid, but `/orgs/myorg/packages/container/myrepo/mycontainer` is a 404)
- I am not sure that `quote` can be reliably called without treating `/` as a safe character, there may be other route options that need `/` to be safe? (I don't think so since we're dealing with url structure and users supplying `/` into any param is almost certainly not meant to be treated as a `/`, which would/could change the url structure)
- I cannot pass in my own quoted string (e.g. `myrepo%2Fmycontainer`) because the `%` will get double encoded
## Github Workflow
Unfortunately I do not have a public repository that I can share with you to reproduce this behavior. However, I can show you an example version of the github workflow that we use to build and subsequently tag and push the images which produces this behavior:
```yaml
name: Example Build Workflow
on:
push:
branches:
- master
pull_request:
branches: ["**"]
jobs:
build-api:
runs-on: ubuntu-latest
env:
image: api
registry: ghcr.io
IS_DEFAULT_BRANCH: ${{ github.ref == 'refs/heads/master' }}
DOCKER_BUILDKIT: 1
steps:
- uses: actions/checkout@v2
- uses: docker/login-action@v1
with:
registry: ${{ env.registry }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Docker Build (api)
run: |
docker build \
--tag ${{env.image}} \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from ubuntu:bionic \
--cache-from ${{env.registry}}/${{github.repository}}/${{env.image}}:pr-${{github.event.pull_request.number}} \
--cache-from ${{env.registry}}/${{github.repository}}/${{env.image}}:${{github.sha}} \
--cache-from ${{env.registry}}/${{github.repository}}/${{env.image}}:latest \
--file ./Dockerfile \
.
- name: Tag & Push (SHA)
run: |
docker tag ${{env.image}} ${{env.registry}}/${{github.repository}}/${{env.image}}:${{github.sha}}
docker push ${{env.registry}}/${{github.repository}}/${{env.image}}:${{github.sha}}
- name: Tag & Push (PR)
if: ${{ ! fromJson(env.IS_DEFAULT_BRANCH) }}
run: |
docker tag ${{env.image}} ${{env.registry}}/${{github.repository}}/${{env.image}}:pr-${{github.event.pull_request.number}}
docker push ${{env.registry}}/${{github.repository}}/${{env.image}}:pr-${{github.event.pull_request.number}}
- name: Tag & Push (latest)
if: ${{ fromJson(env.IS_DEFAULT_BRANCH) }}
run: |
docker tag ${{env.image}} ${{env.registry}}/${{github.repository}}/${{env.image}}:latest
docker push ${{env.registry}}/${{github.repository}}/${{env.image}}:latest
```
## Reproduce with Curl
I am also able to produce the expected behavior and the bad behavior with curl:
```bash
$ curl --head -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/orgs/myorg/packages/container/myrepo%2Fmycontainer
HTTP/2 200
server: GitHub.com
[...]
$ curl --head -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/orgs/myorg/packages/container/myrepo/mycontainer
HTTP/2 404
server: GitHub.com
```
## Print_summary Examples
```python
import os
from ghapi.all import GhApi
from ghapi.core import print_summary
ORGANIZATION = "myorg"
PACKAGE_TYPE = "container"
REPO_NAME = "myrepo"
IMAGE_NAME = "mycontainer"
PACKAGE = f"{REPO_NAME}/{IMAGE_NAME}
API_KEY = os.environ.get("GITHUB_TOKEN")
api = GhApi(org=ORGANIZATION, package_type=PACKAGE_TYPE, token=API_KEY)
api.debug=print_summary
packages = api.packages.get_all_package_versions_for_package_owned_by_org(
org=ORGANIZATION,
package_type=PACKAGE_TYPE,
package_name=PACKAGE
)
api.debug=None
```
```
$ python3 cleanup.py
{'data': None,
'full_url': 'https://api.github.com/orgs/myorg/packages/container/myrepo%5C/mycontainer/versions',
'headers': {'Accept': 'application/vnd.github.v3+json'},
'method': 'GET'}
[...]
fastcore.basics.HTTP404NotFoundError: HTTP Error 404: Not Found
```
### Same Script, urlencode the / myself
```python
PACKAGE = f"{REPO_NAME}%2F{IMAGE_NAME}"
```
```
$ python3 cleanup.py
{'data': None,
'full_url': 'https://api.github.com/orgs/myorg/packages/container/myrepo%252Fmycontainer/versions',
'headers': {'Accept': 'application/vnd.github.v3+json'},
'method': 'GET'}
[...]
fastcore.basics.HTTP404NotFoundError: HTTP Error 404: Not Found
```
### Other Variations I've tried
None of these work either, probably for pretty obvious reasons, but I wanted to show you the things I tried in as complete a detail as I can.
```python
PACKAGE = r"myrepo/mycontainer"
[...]
PACKAGE = r"myrepo%2Fmycontainer"
[...]
PACKAGE = "myrepo\%2Fmycontainer"
```
## Expected behavior
I should be able to call the packages api with packages that have `/` in their identifier, since github allows it and I can curl it.
## Proposed Solutions
### Option 1 - Blanket Change
```python
if route:
for k,v in route.items(): route[k] = quote(str(route[k]), safe='')
```
```python
>>> import urllib.parse
>>> urllib.parse.quote("myrepo/mycontainer")
'myrepo/mycontainer'
>>> urllib.parse.quote("myrepo/mycontainer", safe='')
'myrepo%2Fmycontainer'
```
### Option 2 - Skip Quote Config
```python
if route:
if self.no_quote_routes:
for k,v in route.items(): route[k] = str(route[k])
else:
for k,v in route.items(): route[k] = quote(str(route[k]))
```
### Option 3 - No Safe Quote Config
```python
if route:
if self.no_safe_quote:
for k,v in route.items(): route[k] = quote(str(route[k]), safe='')
else:
for k,v in route.items(): route[k] = quote(str(route[k]))
```
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia tracciando come vengono costruite le route di GhApi e come urllib.parse.quote viene applicato ai parametri delle route, usando l'esempio get_all_package_versions_for_package_owned_by_org e l'output di ghapi.core.print_summary. Verifica che la richiesta per un nome di pacchetto contenente `/` codifichi quella barra come `%2F` esattamente una volta, mentre il restante comportamento delle route rimane corretto.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- api
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100