Automattic / Automattic/jetpack

Search: add indexing for Gutenberg, embeds, and media filtering

Open
#9,054 9 comments 0 reactions 0 assignees View on GitHub
[Feature] Search [Focus] Blocks [Status] Search Index Change Enhancement
Dominant language
PHP
Stars
1.8k
Forks
898
Avg merge
1d 18h
Merged PRs (30d)
774

Description

Gutenberg is around the corner and will have some interesting impacts on searching.

All of our content gets passed through clean_string() (https://github.com/Automattic/wpes-lib/blob/master/src/common/class.wpes-abstract-field-builder.php#L44) and so html tags/comments should get stripped and cleaned up. So Gutenberg shouldn't make anything worse. But there will be lots of meta content that we could do more with.

A big improvement would be to support filtering by blocks. `?s="search terms"&block=video` was suggested. I feel like maybe "block" is not the best name here, but not sure what would be better. I think we want to let end users filter by "media".

We already extract shortcodes, and should probably have something that supports both. So all shortcodes that are video should be able to also get matched along with gutenberg video blocks.

Right now we have:
- has.image, has.gallery, has.shortcode, has.mention, etc - each of these is a count of the number of each of these items in the post
- shortcode..id - just the id field within the shortcode
- shortcode_types - an array of the shortcode_types that were found

This is all implemented with the Jetpack media extractor: https://github.com/Automattic/jetpack/blob/master/_inc/lib/class.media-extractor.php#L95

Something similar could be used for blocks.

Proposal:
1. A couple of new fields that can be used for filtering and indexing. Base the new block fields on what we are doing with shortcodes today.
2. Add a way for end users to filter by "media" where we take the existence of specific blocks/shortcodes and tag the post as having a type of media: pdf,
3. Adjust search query so that we will match against block names by default, but will not boost on those matches very strongly. Also support matching against the "media" on the post. This requires some form of all_content field as discussed in #8895

For the "media" field we should be able to filter that in the sidebar similar to other fields:

![screen shot 2018-03-14 at 10 51 57 am](https://user-images.githubusercontent.com/820871/37417397-d4a26ca4-2775-11e8-8b37-40e5f0207bf1.png)

I am not sure an end user would really want to filter by "block". Feels too granular. A paragraph is a block. Certainly a developer will want to filter by block and so we should have a `block_types` field but that probably won't be exposed in front end search.

@MichaelArestad @jeffgolenski any thoughts on the design here?

Some methods for getting this built:
- We have the demo post with all the blocks: https://gist.github.com/gibrown/ea228eb6db8a783209d89f445c49d6c0
- Should create a wp.com site with one block per post, and some combinations so we can actually test indexing and test out searching against them
- Gutenberg has a grammar for extracting blocks: https://github.com/WordPress/gutenberg/blob/master/blocks/api/post.pegjs and https://automattic.github.io/wp-post-grammar/explorer/
- Going to need some test data. Maybe we should build an exported list of posts similar to the theme test posts. Would help us and others.

Contributor guide

Open the contributing guide

Research direction

Start with Jetpack’s media extractor and the existing shortcode fields, then compare them with Gutenberg’s post grammar and the demo post containing all blocks. Review #8895 for the all_content dependency and determine how block_types, media types, and query matching should work. Done means the indexing, media sidebar filtering, and search behavior are specified and covered with representative test data.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, wordpress
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.