[DOC] comparing_indexes.md / methodologies

Documentation feedback following fern conversion - Methodologies

fern/pages/comparing_indexes.md / methodologies section

This is not urgent for getting fixed in 26.06. I'm reviewing based on the current release 26.06 branch.

Generally this page has excellent content. There are just a few improvements needed to really make it shine.

Overall Page Feedback

The following pieces should be defined early on the page

• recall is the primary quality metric, and when we say quality, we mean recall
• recall bucket
• tuning run

I think the best page structure with the current material is the following, which solves definitional issues along with being a topological sort of concepts:

1. Intro matter as it is, with the addition of something like "recall is the primary quality metric, and when we say quality, we mean recall" and a definition of 'recall bucket' and 'tuning run'.
2. Recall section - this section covers quality, and while it shows Pareto curves, you don't need to understand Pareto curves to understand this, and this never mentions Pareto curves. It ends with a mention of fair comparison which tees up:
3. Fair Comparisons - shows a Pareto curve, but doesn't mention Pareto curve and you don't need to understand Pareto to understand it. This section gives a great explanation and image for a recall bucket, which is used in the Pareto section.
4. Pareto Curves - now we are ready to understand these
5. Large Datasets
6. Methodology summary

If you do make these structural changes, also change this sentence in the intro in order to fit the new structure and to use consistent terminology ('fair comparisons' instead of 'consistent reporting') from:

"This page describes how to make benchmark results comparable by using recall buckets, Pareto curves, and consistent reporting for build and search metrics."

to

"This page describes how to make benchmark results comparable by using recall buckets, fair comparisons, and Pareto curves for build and search metrics."

I will now review the remaining sections in my suggested reordering.

Recall

"Recall measures how many exact nearest neighbors were returned by an approximate search. "

I would add onto this sentence "for a given value of k"

Fair Comparisons

This section is the gold standard for this page. Clear, simple, good tables and supporting figures, consistent usage.

Pareto Curves

This section is super important and IMO not clear enough.

Imagine every tuning run is a toy car. You want a car that is fast, but you also care how much work it took to build. If one car is both faster and easier to build than another car, the slower and harder-to-build car is not a useful choice. The cars that are not beaten this way form the Pareto curve.

1. Tuning run should be defined before this (ideally in the intro).
2. This metaphor was good but the last sentence ('The cars that are not beaten this way form the Pareto curve.') was confusing (what does it mean to be not beaten this way?) and didn't complete the idea for me. A car that is faster AND easier than other cars goes on the curve. I think a car that is faster OR easier than other cars goes on the curve, but I'm not exactly sure.

For vector indexes, each tuning run is a point with quality, build time, and search performance. A point is on the Pareto curve when no other run is better on the metric being compared without making another metric worse. Finding these points usually requires a parameter sweep or another hyperparameter optimization method.

1. I think an example can really help here. 'A point is on the Pareto curve when no other run is better on the metric being compared without making another metric worse. For example, if a Pareto point for a tuning run has a given value of latency and recall, another tuning run will only be a Pareto point if it has lower latency, even if it has worse recall.' Or whatever is the exact right phrasing.
2. Dominated run is used in the figure but not defined. This goes along with the above explanation - I think that a dominated point is one that does not appear on the Pareto curve, but it should be defined.
3. It wasn't clear to me why some points were dominated. E.g., second gray point from the left is better on recall but worse on latency than the very first leftmost green point on Pareto curve, but it is dominated. Is there something about a point not just being strictly better than another point, but somehow being worse than the line of interpolation between Pareto points? If so, this should be explained.

For each quality bucket, summarize build time by taking the points on the Pareto curve in that bucket and averaging their corresponding build times. This gives an expected build time for the quality window instead of forcing one run to represent the whole bucket.

1. Elsewhere on the page we use 'recall bucket'. I suggest replacing both 'quality bucket' and 'quality window' with 'recall bucket', otherwise those are new undefined terms.

Large Datasets

Representative-sample tuning is appropriate when the benchmarked sample matches the unit that will actually be searched in production. For locally partitioned systems, that usually means tuning against the expected partition or segment size, not the full database size.

This section feels like advanced content, there are a bunch of new terms introduced. That's probably fine, might be worth pointing out.

1. If nothing else, define "Representative-sample tuning" and explain what "the unit that will actually be searched in production" means.

Start with fern/pages/comparing_indexes.md, especially the methodologies section, and review the current section order and terminology. Reorganize the page around Recall, Fair Comparisons, Pareto Curves, Large Datasets, and the methodology summary; define recall, recall bucket, tuning run, dominated run, and representative-sample tuning, and apply the requested wording changes. Done means the suggested structure and definitions are reflected consistently throughout the page.