psf / psf/cachecontrol

Not possible to make an heuristic depend on the request.

Open
#131 4 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
505
Forks
149
Avg merge
2d 13h
Merged PRs (30d)
7

Description

We're having a bit of trouble with an heuristic. The fact is, Heuristic classes get the raw response, (a.k.a the requests.packages.urllib3.response.HTTPResponse), which does not contain any information about the request (for example, URL ...).

If we look at the parent's build_response(...) method, we see it has both raw request and response (cache_response method takes both).

    def build_response(self, request, response, from_cache=False):
        """
        Build a response by making a request or using the cache.

        This will end up calling send and returning a potentially
        cached response
        """
        if not from_cache and request.method == 'GET':

            # apply any expiration heuristics
            if response.status == 304:
                # We must have sent an ETag request. This could mean
                # that we've been expired already or that we simply
                # have an etag. In either case, we want to try and
                # update the cache if that is the case.
                cached_response = self.controller.update_cached_response(
                    request, response
                )

                if cached_response is not response:
                    from_cache = True

                # We are done with the server response, read a
                # possible response body (compliant servers will
                # not return one, but we cannot be 100% sure) and
                # release the connection back to the pool.
                response.read(decode_content=False)
                response.release_conn()

                response = cached_response

            # We always cache the 301 responses
            elif response.status == 301:
                self.controller.cache_response(request, response)
            else:
                # Check for any heuristics that might update headers
                # before trying to cache.
                if self.heuristic:
                    response = self.heuristic.apply(response)

                # Wrap the response file with a wrapper that will cache the
                #   response when the stream has been consumed.
                response._fp = CallbackFileWrapper(
                    response._fp,
                    functools.partial(
                        self.controller.cache_response,
                        request,
                        response,
                    )
                )

Is there any reason (other than BC, obviously) why the heuristic only gets then response, and not the request ?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the parent build_response(...) method and the heuristic.apply(response) call shown in the issue. Trace how the request and raw response reach this point, then determine the request-aware heuristic interface and compatibility requirements; done means the needed request information is available to heuristics without breaking existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.