race condition when deleting a repository and associated distribution

Author: @jlsherrill (jsherril@redhat.com)

Redmine Issue: 8942, https://pulp.plan.io/issues/8942

In our katello tests, we have some 'cleanup' code that:

list and then deletes a repository with a given name
list and then deletes a remote with a given name
list and then deletes a distribution with a given name
wait on all the tasks generated from above

This has worked for quite a while, but after upgrading from pulp-container 2.3 to 2.6, we started seeing an error (ISE) from the server:

LookupError: view not found

when trying to list the distributions on the 3rd step.

I'm not sure if the pulp code is running faster, or slower, but it appears that there is a consistent race condition occuring. I can modify the code to poll the tasks after the repo delete, and it seems to work find (so this isn't blocking us)

Here's the full traceback:

Untitled
From WebUI, 1 Second ago, written in Plain Text.
This paste will self destruct in 1 Day.	
URL https://paste.centos.org/view/99464b52
Embed Show code
Download Paste or View Raw

        Jun 21 16:14:09 devel2 pulpcore-api: pulp [b51142c5bced40989a59403cf6e92fd8]: django.request:ERROR: Internal Server Error: /pulp/api/v3/distributions/container/container/
        Jun 21 16:14:09 devel2 pulpcore-api: Traceback (most recent call last):
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
        Jun 21 16:14:09 devel2 pulpcore-api: response = get_response(request)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
        Jun 21 16:14:09 devel2 pulpcore-api: response = self.process_exception_by_middleware(e, request)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
        Jun 21 16:14:09 devel2 pulpcore-api: response = wrapped_callback(request, *callback_args, **callback_kwargs)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
        Jun 21 16:14:09 devel2 pulpcore-api: return view_func(*args, **kwargs)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 125, in view
        Jun 21 16:14:09 devel2 pulpcore-api: return self.dispatch(request, *args, **kwargs)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 509, in dispatch
        Jun 21 16:14:09 devel2 pulpcore-api: response = self.handle_exception(exc)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 469, in handle_exception
        Jun 21 16:14:09 devel2 pulpcore-api: self.raise_uncaught_exception(exc)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
        Jun 21 16:14:09 devel2 pulpcore-api: raise exc
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 506, in dispatch
        Jun 21 16:14:09 devel2 pulpcore-api: response = handler(request, *args, **kwargs)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/mixins.py", line 43, in list
        Jun 21 16:14:09 devel2 pulpcore-api: return self.get_paginated_response(serializer.data)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 745, in data
        Jun 21 16:14:09 devel2 pulpcore-api: ret = super().data
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 246, in data
        Jun 21 16:14:09 devel2 pulpcore-api: self._data = self.to_representation(self.instance)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 664, in to_representation
        Jun 21 16:14:09 devel2 pulpcore-api: self.child.to_representation(item) for item in iterable
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 664, in <listcomp>
        Jun 21 16:14:09 devel2 pulpcore-api: self.child.to_representation(item) for item in iterable
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/serializers.py", line 515, in to_representation
        Jun 21 16:14:09 devel2 pulpcore-api: ret[field.field_name] = field.to_representation(attribute)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/rest_framework/relations.py", line 401, in to_representation
        Jun 21 16:14:09 devel2 pulpcore-api: url = self.get_url(value, self.view_name, request, format)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/fields.py", line 210, in get_url
        Jun 21 16:14:09 devel2 pulpcore-api: repo_url = rvr_field.get_url(obj.repository, None, request, *args, **kwargs)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/fields.py", line 198, in get_url
        Jun 21 16:14:09 devel2 pulpcore-api: return super().get_url(obj, self.view_name, request, *args, **kwargs) + "versions/"
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/base.py", line 236, in get_url
        Jun 21 16:14:09 devel2 pulpcore-api: view_name = self._view_name(obj)
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/pulpcore/app/serializers/base.py", line 231, in _view_name
        Jun 21 16:14:09 devel2 pulpcore-api: return get_view_name_for_model(obj, "detail")
        Jun 21 16:14:09 devel2 pulpcore-api: File "/usr/lib/python3.6/site-packages/pulpcore/app/util.py", line 78, in get_view_name_for_model
        Jun 21 16:14:09 devel2 pulpcore-api: raise LookupError("view not found")
        Jun 21 16:14:09 devel2 pulpcore-api: LookupError: view not found
        Jun 21 16:14:09 devel2 pulpcore-api: pulp [b51142c5bced40989a59403cf6e92fd8]:  - - [21/Jun/2021:16:14:09 +0000] "GET /pulp/api/v3/distributions/container/container/?name=Default_Organization-Test-busybox-library HTTP/1.1" 500 27 "-" "OpenAPI-Generator/2.4.0/ruby"

Start at pulpcore/app/serializers/fields.py and pulpcore/app/util.py, following the distribution list endpoint shown in the traceback. Reproduce the repository, remote, and distribution cleanup sequence without waiting between tasks, then inspect how serialization reaches get_view_name_for_model. Done means the distribution listing no longer returns a 500 or LookupError during that sequence.