CKAN Coding Standards -- Dependencies

**CKAN version**

All currently shipping

**Describe the bug**

The current CKAN coding standards for extensions leads to inconsistent installs and works against the zen of the python packaging system.

Specifically: (https://docs.ckan.org/en/2.9/extensions/best-practices.html#add-third-party-libraries-to-requirements-txt)
> If your extension requires third party libraries, rather than adding them to setup.py, they should be added to requirements.txt, which can be installed with: `pip install -r requirements.txt`

Separating the dependencies from setup.py breaks the pip resolver. So not only are we likely to get conflicts, we can't even use the tools to figure out what they are.

**Steps to reproduce**

A small example:
* Install ckan. A pinned version of redis is installed.
* Install ckanext-harvest. The original version of redis is uninstalled and a different version of redis is installed.

I'm sure that there are other examples, e.g requests, six.

**Expected behavior**

Ideally, the ckan, extensions, and all of the python dependencies could be installed with: `pip install --use-feature=2020-resolver ckan ckanext_foo ckanext_bar ckanext_scheming ckanext_pages...`

This would be a one-shot install, and the pip resolver could build the entire dependency tree, and inform us of conflicts, which could then be resolved.

Then, we could see the results of our dependencies with `pipdeptree`:

```
$ pipdeptree
Warning!! Cyclic dependencies found:
* SPARQLWrapper => rdflib => SPARQLWrapper
* rdflib => SPARQLWrapper => rdflib
------------------------------------------------------------------------
bleach==2.1.3
- html5lib [required: >=0.99999999pre,!=1.0b8,!=1.0b7,!=1.0b6,!=1.0b5,!=1.0b4,!=1.0b3,!=1.0b2,!=1.0b1, installed: 1.0.1]
- six [required: >=1.9, installed: 1.15.0]
- webencodings [required: Any, installed: 0.5.1]
- six [required: Any, installed: 1.15.0]
ckan==2.8.7
ckanext-dcat==1.1.1
ckanext-harvest==1.3.2
- ckantoolkit [required: ==0.0.3, installed: 0.0.3]
- pika [required: >=1.1.0, installed: 1.2.0]
- redis [required: <=3.0, installed: 2.10.6]
- requests [required: >=2.11.1, installed: 2.25.1]
- certifi [required: >=2017.4.17, installed: 2020.12.5]
- chardet [required: >=3.0.2,<5, installed: 4.0.0]
- idna [required: >=2.5,<3, installed: 2.10]
- urllib3 [required: >=1.21.1,<1.27, installed: 1.26.3]
- six [required: >=1.12.0, installed: 1.15.0]
...
```

Where now, CKAN has no dependencies. (note, ckanext-harvest has had it's requirements hoisted (and tweaked) into setup.py in my install.)

**Additional details**

To reduce the number of conflicts, extensions should be liberal in the versions what dependencies they pin, e.g. pinning a specific version of requests is unnecessary, especially given how stable the requests interface is.

```
requests>=2,<3
```

Obviously, there are cases where a specific pinned dependency is required. That's to be expected if you are relying on the specific behavior of a dependency Transitive dependencies should not be pinned. (e.g., certifi and pytz have stable interfaces, and should generally _always_ be on the latest version). Any extension dependency that's also a dependency of CKAN shouldn't be pinned, because then the CKAN pins for different major versions will conflict.

CKAN should be the most specific pin, and extensions compatible with a specific version of CKAN should be compatible with that pin. However, CKAN should also be generally forgiving of the exact version of dependencies that have relatively stable interfaces where it's not using 'deep' features.

Ideally, CKAN would also be only pinning direct dependencies to make it easier to have a clear view of what needs to be updated on point releases. But that's a further bridge to cross.

Start with the CKAN extension best-practices page linked in the issue, especially its third-party library guidance, and review the documented pip install workflow. Reproduce the dependency conflict with CKAN and ckanext-harvest, then inspect the resulting pipdeptree output; done means the standards clearly support resolver-compatible dependency installation without the reported conflicts.

Written by the indexing model from the issue text.