ckan / ckan/ckanext-spatial

Solr queries not working due to local parameters not being allowed since CKAN 2.10.4/2.9.11

Open
#328 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
XSLT
Stars
136
Forks
207
PR merge metrics
No merged PRs in 30d

Description

## Environment
- CKAN 2.9.11 (custom `ckan-docker`)
- Custom `ckan-solr` with spatial config for `solr-bbox` and `solr-spatial-field` backends.
- `ckanext-spatial`

## Problem:
- Spatial searches with the Solr backend `solr-bbox` and `solr-bbox` using the `{!frange}` local parameter.
- Local parameters in Solr currently do not work in [`ckan-2.9.11`](https://github.com/ckan/ckan/blob/master/CHANGELOG.rst#minor-changes-3) due to security restrictions [disallowing local parameters](https://github.com/ckan/ckan/pull/8053).
- The error message encountered are:

**solr-bbox** log:
```log
2024-04-12 13:26:32,813 INFO [ckan.lib.search.query] SearchError query: {'facet.field': ['theme', 'groups', 'theme_es', 'dcat_type', 'publisher_name', 'publisher_type', 'spatial_uri', 'owner_org', 'res_format', 'frequency', 'tags', 'tag_uri', 'conforms_to'], 'fq': [' -dataset_type:harvest', '{!frange incl=false l=0 u=1}div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))', '+site_id:"default"', '+state:active'], 'q': '*:*', 'rows': 21, 'sort': 'score desc, metadata_modified desc', 'start': 0, 'df': 'text', 'fq_list': ['{!frange incl=false l=0 u=1}div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))'], 'bf': 'div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))', 'defType': 'edismax', 'fl': 'id validated_data_dict', 'facet': 'true', 'facet.limit': '50', 'facet.mincount': 1, 'wt': 'json', 'tie': '0.1', 'mm': '2<-1 5<80%', 'qf': 'name^4 title^4 tags^2 groups^2 text', 'q.op': 'AND'}
2024-04-12 13:26:32,813 ERROR [ckan.views.dataset] Dataset search error: ("Local parameters are not supported in param 'fq'. value: '{!frange incl=false l=0 u=1}div(mul(mul(max(0,sub(min(7.910156250000001,maxx),max(-29.355468750000004,minx))),max(0,sub(min(45.91485503041036,maxy),max(23.193387206172922,miny)))),2),add(846.7296993875983,mul(sub(maxy,miny),sub(maxx,minx))))'",)
```

**solr-spatial-field** log:
```log
2024-04-12 13:05:20,561 INFO [ckan.lib.search.query] SearchError query: {'facet.field': ['theme', 'groups', 'theme_es', 'dcat_type', 'publisher_name', 'publisher_type', 'spatial_uri', 'owner_org', 'res_format', 'frequency', 'tags', 'tag_uri', 'conforms_to'], 'fq': [' -dataset_type:harvest', '{!field f=spatial_geom}Intersects(ENVELOPE(-29.355468750000004, 7.910156250000001, 45.91485503041036, 23.193387206172922))', '+site_id:"default"', '+state:active'], 'q': '*:*', 'rows': 21, 'sort': 'score desc, metadata_modified desc', 'start': 0, 'df': 'text', 'fq_list': ['{!field f=spatial_geom}Intersects(ENVELOPE(-29.355468750000004, 7.910156250000001, 45.91485503041036, 23.193387206172922))'], 'fl': 'id validated_data_dict', 'facet': 'true', 'facet.limit': '50', 'facet.mincount': 1, 'wt': 'json', 'q.op': 'AND'}
2024-04-12 13:05:20,561 ERROR [ckan.views.dataset] Dataset search error: ("Local parameters are not supported in param 'fq'. value: '{!field f=spatial_geom}Intersects(ENVELOPE(-29.355468750000004, 7.910156250000001, 45.91485503041036, 23.193387206172922))'",)
```

![image](https://github.com/ckan/ckanext-spatial/assets/96422458/45f47e6f-48c4-4765-9905-25c111201f7c)

## Reference the solution:
CKAN pull request https://github.com/ckan/ckan/pull/8053 introduces a new configuration option `ckan.search.solr_allowed_query_parsers` which allows to specify whitelisted local parameters. If the search parameters are not included, this will cause some problems in `ckanext-spatial` like `{!frange}`:

https://github.com/ckan/ckanext-spatial/blob/938308469892e4bcf7389cb4adee5ccdd5a0ccca/ckanext/spatial/search/__init__.py#L123-L124

or `{!field}`:

https://github.com/ckan/ckanext-spatial/blob/938308469892e4bcf7389cb4adee5ccdd5a0ccca/ckanext/spatial/search/__init__.py#L202-L211

## Proposed fix:
Updated the doc to add params to the `ckan.search.solr_allowed_query_parsers` list in the `ckan.ini` configuration file when setting up spatial search.

```rst
Setup
-----

To enable the spatial search you need to add the ``spatial_query`` plugin to
your ini file. This plugin in turn requires the ``spatial_metadata`` plugin, eg::

ckan.plugins = ... spatial_metadata spatial_query

To define which backend to use for the spatial search use the following
configuration option (see `Choosing a backend for the spatial search`_)::

ckanext.spatial.search_backend = solr-bbox

For CKAN >2.10.4 and >2.9.11 to avoid errors in the spatial search, use the following configuration elements. For `solr-bbox` use `frange`, for `solr-spatial-field` use `field`::

ckan.search.solr_allowed_query_parsers = ... frange | field

```

![image](https://github.com/ckan/ckanext-spatial/assets/96422458/3c5dfee4-7f48-4305-aa1a-049f9fac4b70)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the spatial search entry points in ckanext/spatial/search/__init__.py, especially the referenced lines that emit the `frange` and `field` local parameters. Update the spatial-search setup documentation to explain the matching `ckan.ini` setting, then verify that the instructions distinguish `solr-bbox` from `solr-spatial-field` and include the required parser names.

Written by the indexing model from the issue text.

Assessment

Domain
documentation, search
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.