Proposal: Help search engines find latest version of a crate

Motivation and Summary

When someone uses a web search engine to find a crate’s documentation, they are likely to wind up on the documentation for a random older version of that crate. This can be confusing and frustrating if they are working on a project that depends on a more recent version of that crate. As an example: in April 2021, a Google search for [rustls serversession] links to version 0.5.5 of that crate, released Feb 2017. A Google search for [rustls clientsession] links to version 0.11.0, released Jan 2019. The latest version is 0.19.0, released Nov 2020.

To fix this, I propose that doc.rs’s URL structure should be more like crates.io: Each crate should have an unversioned URL (docs.rs/rustls/latest) that always shows the docs for the latest version of that crate. There would continue to be versioned URLs like today (https://docs.rs/rustls/0.19.0/rustls/), accessible as defined below. I believe this will, over time, lead search engines to more often find the unversioned URL.

This is a popular request:

https://github.com/rust-lang/docs.rs/issues/1006
https://github.com/rust-lang/docs.rs/issues/854
https://github.com/rust-lang/docs.rs/issues/74
https://github.com/rust-lang/docs.rs/issues/1411

It's also a problem that disproportionately affects new users of the language who haven't gotten in the habit of looking for the "go to latest version" link. I know when I was first learning Rust, this problem was a particular headache for me.

Non-working solutions

<link rel=canonical> is a commonly proposed solution, but it’s not the right fit:

The target (canonical) IRI MUST identify content that is either
duplicative or a superset of the content at the context (referring)
IRI.

Since documentation of different versions is not duplicative, this won’t work. And in fact search engines verify the property, and will disregard canonical links on a site if it does not hold.

Here are some links about Google’s handling of canonical:

https://developers.google.com/search/docs/advanced/guidelines/duplicate-content
https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls

Proposed Solution

For any given crate, https://docs.rs/<crate>/latest should exist and not be a redirect. It should serve the latest version of that crate. Crates.io should be updated so that the unversioned URL for a crate (e.g. https://crates.io/crates/ureq) links to the unversioned URL on docs.rs.

Sometimes people will want to link to a specific version of the documentation rather than the generic “latest” URL. There will be two ways to do that:

• You can navigate explicitly to the versioned URL, using the version selector tools already available in docs.rs.
• Additionally, we should follow GitHub’s UI for when you click on a line number in a source file. When you click on an anchor within a doc page, a little “...” should appear with options to either copy the line or copy a permalink. We may even want to mimic the keystroke to go to a permalink for a page (y).

Caching issues

Currently, only static files are cached. The things that change between versions of a crate are its HTML and some JS (containing JSON data used in displaying the pages). The HTML is currently not cached at all, so invalidating its cache is not a current concern. The JS is also not cached, but it has a unique URL per crate version so easily could be cached.

In case we later decide to start caching the HTML: The CloudFront rates for cache invalidation are reasonable: $0.005 per invalidation request, and purging a whole subdirectory (like /crates/<crate>/*) is considered a single invalidation request.

Will it work?

I’m pretty sure it will work. Search engines these days take navigation events heavily into account, so if most navigation events go to an unversioned URL, that will help a lot. Also, once we make this change, the unversioned URLs will start accumulating “link juice,” which will also help a lot.

One good demonstration that it will work is that crates.io already follows a scheme like this, and does not have the "links go to old versions" problem at all.

Start by tracing how docs.rs currently resolves versioned crate URLs and how crates.io generates its documentation link. Define the routing and linking changes needed for //latest to serve the newest version without redirecting, while preserving versioned URLs; consider the proposed permalink behavior separately. Done means the latest URL works reliably and crates.io points to it.