internetarchive / internetarchive/openlibrary

Search: requesting `editions` field silently filters out works that match only via work-level fields (series, subject, etc.)

Open
#12,688 1 comment 0 reactions 1 assignee Claimed by @cdrini View on GitHub
Lead: @cdrini Module: Solr Needs: Staff Decision Needs: Triage Theme: Search Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

When a `/search.json` request includes `editions` in the `fields` parameter (e.g. `fields=key,title,editions`), the worksearch backend rewrites the work query to **additionally require a matching edition**. This silently discards works that match the user query only via work-level Solr fields (`series_name`, `subject_facet`, `person_facet`, `place_facet`, `time_facet`, etc.).

The query and result count drop with no error, no warning header, and nothing in the response distinguishing "I really had 0 results" from "I had results but the editions filter killed them."

This is intentional in the code (with the comment *"The elements in this edition query should cause works not to match at all if matching editions are not found"*), but the implementation conflates "matches the user query" with "has an edition that matches the user query," which is wrong for any query that targets work-only metadata.

**Concrete impact on production:**

```
q=narnia, fields=key → 853 results
q=narnia, fields=key,editions → 790 results (-63, -7.4%)
```

Same query against a smaller / less-rich index (local dev) — total loss:

```
q=narnia, fields=key → 7 results
q=narnia, fields=key,editions → 0 results (every Narnia book disappears)
```

The legacy header-bar autocomplete (`openlibrary/plugins/openlibrary/js/SearchBar.js`) requests `editions` in its `DEFAULT_JSON_FIELDS`, so production users typing series names into the header search are quietly missing matching books and don't realize it.

### Reproducing the bug

1. Hit `https://openlibrary.org/search.json?q=narnia&limit=1&fields=key`. Note `numFound: 853`.
2. Hit `https://openlibrary.org/search.json?q=narnia&limit=1&fields=key,editions`. Note `numFound: 790`.

The 63-work delta is composed entirely of works whose only match against the query "narnia" was via `series_name` (The Chronicles of Narnia) and which don't have an edition with "narnia" in any indexed text field.

* **Expected behavior:** Including `editions` in `fields` should add an `editions` block to each result; it should not change which works appear or how many `numFound` is.
* **Actual behavior:** Adding `editions` causes works to be filtered out post-match. Same query, same data, different `numFound`.

Other reproducer queries that match work-only fields:
* `q=fantasy` (matches via `subject_facet`)
* `q=napoleon` (matches via `person_facet`)
* `q=paris` (matches via `place_facet`)

Author and title queries (e.g. `q=tolkien`, `q=harry potter`) usually survive because those terms are also indexed on editions.

### Context

* Browser (Chrome, Safari, Firefox, etc): n/a — backend behavior, reproducible via curl
* OS (Windows, Mac, etc): n/a
* Logged in (Y/N): N (anonymous)
* Environment (prod, dev, local): both production and local dev (more visible on dev due to smaller index)

### Breakdown

Implementation Details (for maintainers)

The filter is in `openlibrary/plugins/worksearch/schemes/works.py` lines 555–572, in `WorkSearchScheme.process_user_query()`. When `editions:[subquery]` is present in `solr_fields`, the work query is rewritten:

```python
if ed_q or len(editions_fq) > 1:
# The elements in _this_ edition query should cause works not to
# match _at all_ if matching editions are not found
q = (
f'+{full_work_query} '
'+('
'_query_:\"{!parent which=type:work v=\$fullEdQuery filters=\$editions.fq}\" '
'OR edition_count:0'
')'
)
```

The `+(...)` clause is a required AND. The parent block-join requires at least one matching edition; the `OR edition_count:0` escape hatch only saves edition-less works. Works with editions but no edition matching the query are dropped.

Why it's hard to notice in production:
- `numFound` reflects the post-filter count, so there's no "I lost results" signal in the response.
- The legacy autocomplete renders ~10 results — a 7% loss usually means losing the bottom-ranked works.
- On a dense index (production), most works do have an edition with overlapping text (e.g. an edition with a subtitle that mentions the series), so the loss looks like ranking drift rather than lost results.

#### Requirements Checklist

- [ ] Pick a fix direction (suggested options below)
- [ ] Update `WorkSearchScheme.process_user_query()` accordingly
- [ ] Document the chosen behavior in the search API docs
- [ ] Confirm legacy callers (notably `SearchBar.js` `DEFAULT_JSON_FIELDS`) get expected results post-fix

#### Reasonable fix directions

1. **Decouple "include editions data" from "filter works by editions"** — add a separate flag (e.g. `?require_matching_edition=true`) so callers opt into the filter explicitly. Default off; let the cover-fallback callers omit it. Cleanest of the three because it makes today's behavior an opt-in instead of a side effect.
2. **Expand the `OR edition_count:0` escape clause** to also OR-in works whose query terms are work-only fields (series, subject, etc.), since by definition no edition could match those.
3. **Document the behavior** in the API docs at minimum, so callers stop being surprised. Lowest effort, doesn't fix the bug.

#### Related files

* \`openlibrary/plugins/worksearch/schemes/works.py\` — `process_user_query()` lines 555–572 (the filter), 596–622 (the editions subquery setup)
* \`openlibrary/plugins/openlibrary/js/SearchBar.js\` — \`DEFAULT_JSON_FIELDS\` includes \`editions\`; legacy header autocomplete is one of the affected callers

#### Stakeholders

* @cdrini


#### Instructions for Contributors

- Please [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.