Automattic / Automattic/jetpack

Search: Index pdfs for searching

Open
#9,023 7 comments 0 reactions 0 assignees View on GitHub
[Feature] Search [Status] Search Index Change Customer Report [LEGACY] Enhancement
Dominant language
PHP
Stars
1.8k
Forks
898
Avg merge
1d 18h
Merged PRs (30d)
774

Description

We have support for indexing pdfs implemented for VIP sites and there have been a few requests for it for general Jetpack Search. The indexing is here: https://github.com/Automattic/wpes-lib/blob/master/src/common/class.wpes-wp-post-field-builder.php#L671

Open questions:
1. Can we lower that 10MB limit on file size low enough? There have been some hiccups with large files and that is even with the restriction that the files are hosted on vip infrastructure, what happens when we are fetching them from external sites on demand? How do we make sure we don't DDoS a site?
2. Should we support other doc formats? Which ones?
3. How should search results get displayed? For VIP we added fields as stored fields:

- has.link_internal (just a count of internal links)
- link_internal.post_id, link_internal.post_type, link_internal.comment_id (we already have link.url).

We then return these fields when we run this sort of search and it let’s the client display some number of them beneath each result. So we can match the pdf as it's own document, basically this is like a media post search which includes the content of the pdf. But we cannot match a post/page that has a pdf attached to it.

Integration with themes may get tricky because of this. VIPs seem to have not had too many issues, but not very many have used it yet.

Contributor guide

Open the contributing guide

Research direction

Start with the PDF indexing implementation in wpes-lib/src/common/class.wpes-wp-post-field-builder.php around line 671. Resolve the file-size and fetching safeguards, supported document formats, search-result representation, and theme integration questions before defining the implementation scope; done requires an agreed design and follow-up changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, wordpress
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.