Spike: how to improve the docs search functionality

Why are we doing this?

Large numbers of users have complained that the search functionality in the OS Docs is not effective:

• users are unable to find the information they're looking for (this maybe an issue of mismatched semantics)
• often searches lead users to archived docs, which they're not expecting so they're either annoyed or confused. "I kept getting cohort extractor stuff"
• currently BI aren't aware of what searches users are making, it would be good to be able to improve docs (content or structure) in the future based on this insight
• there have been reports that users often use another search engine to find the information they're looking for

Although recent exploration suggests this is not the main issue, a lot of users are asking for it to be improved and it would be remiss of us to ignore this. Additionally, it is an experiment for us to understand how much we can improve user experience with improving the search. It maybe that the underlying structure of OS docs needs further work still.

How will we know when it's done?

We will have a clearer idea of which approach is better for our needs: how much of the above issues might be solved with alternative tools? MKdocs, algolia? LLM has been mentioned by a couple of folk in the Research team too

What are we doing?

timebox?

Note: this was originally create here.

Defining delivery tasks guidance

Start by reviewing the linked exploration and the original job-server issue, then document the current OS Docs search behavior and the reported failure cases. Compare the mentioned alternatives, including MkDocs, Algolia, and an LLM approach; done means recommending a clearer direction and identifying the delivery tasks needed.