AlmaLinux / AlmaLinux/build-system

Add the ability in bootstrap_repositories.py to resume the process from where it stopped

Open
#265 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
32
Forks
11
PR merge metrics
No merged PRs in 30d

Description

Tried to import the repos into pulp using the bootstrap_repositories.py scrip and at some point, it failed with the following error

```
INFO:repo-bootstrapper:Syncing Repository: almalinux-8-nfv i686 http://pulp/pulp/content/prod/almalinux-8-nfv-i686/ from RepositoryRemote: almalinux-8-nfv-i686 i686 http://rsync.repo.almalinux.org/almalinux/8/NFV/i686/os/...
Traceback (most recent call last):
File "/code/scripts/bootstrap_repositories.py", line 273, in
main()
File "/code/scripts/bootstrap_repositories.py", line 262, in main
sync(pulp_client.sync_rpm_repo_from_remote(
File "/usr/lib64/python3.9/functools.py", line 888, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
File "/code/env/lib/python3.9/site-packages/syncer.py", line 33, in sync_co
return asyncio.get_event_loop().run_until_complete(co)
File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/code/alws/utils/pulp_client.py", line 500, in sync_rpm_repo_from_remote
result = await self.wait_for_task(task['task'])
File "/code/alws/utils/pulp_client.py", line 615, in wait_for_task
raise Exception(f'Task {str(task)} has failed')
Exception: Task {'pulp_href': '/pulp/api/v3/tasks/dcc80ace-f3ea-419e-9152-6aa7de768fe1/', 'pulp_created': '2022-07-15T09:17:29.108353Z', 'state': 'failed', 'name': 'pulp_rpm.app.tasks.synchronizing.synchronize', 'logging_cid': '97c71e1392b54b808348f7979d9efd93', 'started_at': '2022-07-15T09:17:29.224114Z', 'finished_at': '2022-07-15T09:17:29.819785Z', 'error': {'traceback': ' File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 410, in _perform_task\n result = func(*args, **kwargs)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 464, in synchronize\n remote_url = fetch_remote_url(remote, url)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 298, in fetch_remote_url\n remote_url = fetch_mirror(remote)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 250, in fetch_mirror\n result = downloader.fetch()\n File "/usr/local/lib/python3.8/site-packages/pulpcore/download/base.py", line 187, in fetch\n return done.pop().result()\n File "/usr/local/lib/python3.8/site-packages/pulpcore/download/http.py", line 271, in run\n return await download_wrapper()\n File "/usr/local/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry\n ret = await target(*args, **kwargs)\n File "/usr/local/lib/python3.8/site-packages/pulpcore/download/http.py", line 256, in download_wrapper\n return await self._run(extra_data=extra_data)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run\n self.raise_for_status(response)\n File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status\n response.raise_for_status()\n File "/usr/local/lib64/python3.8/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status\n raise ClientResponseError(\n', 'description': "404, message='Not Found', url=URL('http://rsync.repo.almalinux.org/almalinux/8/NFV/i686/os')"}, 'worker': '/pulp/api/v3/workers/a629915a-b048-4b35-8198-11baa7659196/', 'parent_task': None, 'child_tasks': [], 'task_group': None, 'progress_reports': [], 'created_resources': [], 'reserved_resources_record': ['/pulp/api/v3/repositories/rpm/rpm/01649371-8733-4f17-b2e2-7b9c2f40f17d/', 'shared:/pulp/api/v3/remotes/rpm/rpm/f2eb53aa-6842-4b9f-90a2-62b770779fee/']} has failed
```

The problem was already fixed upstream. Had to run the script again, which takes some time because it starts all over again. It would be great if we can bring to this script the ability to resume from where it failed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.