ArchiveBox / ArchiveBox/ArchiveBox

Enhancement: Use the same URL layout as Archive.org for viewing ArchiveBox Snapshots `https://archive.org/web/<URL>`

Open
#1,085 3 comments 3 reactions 0 assignees View on GitHub
size: medium status: wip touches: API/CLI/Spec touches: docs touches: views/replayers/html/css why: functionality
Dominant language
Python
Stars
28.4k
Forks
1.6k
Avg merge
7h 54m
Merged PRs (30d)
21

Description

To visit an archived version of a website (or archive it automatically) on Archive.org, one can just visit `http://web.archive.org/web/https://example.com/` and it will redirect to `http://web.archive.org/web/20230116145642/https://example.com/` (or whatever the most recent snapshot timestamp is).

To really emobdy the tagline "ArchiveBox is a self-hosted version of archive.org" we should properly support their URL scheme too.

e.g.

- `https://demo.archivebox.io/web/https://example.com` should redirect to the most recent snapshot `https://demo.archivebox.io/web/20230116145642/https://example.com`
- note: support both the ArchiveBox-style timestamp in unix timestamp format e.g. `1673919713` or the Archive.org-style `20230116145642` format and truncated forms `2023`, `202301`, `20230116`
- note: also support visiting using snapshots using [ulid](https://github.com/ulid/spec) uuid instead of timestamp as slug, e.g. `https://demo.archivebox.io/01ARZ3NDEKTSV4RRFFQ69G5FAV/...`
- note: support auto prefix-matching slugs so that `2023` matches `202301`, `20230116`, `20230116145642` automatically, and `01AN4Z07BY` matches `01AN4Z07BY79KA1307SR9X4MV3 ` automatically


Full spec:

`https://demo.archivebox.io/web/` where `SLUG` can be:
- an original URL, with or without scheme, e.g. `https://example.com/index.html`, 'example.com/index.html' ➡️ redirect to most recent snapshot for `https://demo.archivebox.io/web/20230116145642/https://example.com/index.html`
- an ArchiveBox snapshot UUID in [`ulid/spec`](https://github.com/ArchiveBox/ArchiveBox/issues/74) format `01AN4Z07BY79KA1307SR9X4MV3/index.html` or timestamp prefix `01AN4Z07BY/index.html` ➡️ redirect to that exact snapshot `https://demo.archivebox.io/web/20230116145642/https://example.com/index.html`
- an ArchiveBox snapshot timestamp in `YYMMDDHHMMSS`, shortened forms like `YYYYMM`, or unix timestamp format e.g. `20230116145642/index.html` or `202301161456/index.html`, `202301/index.html`, `1673919713/index.html` ➡️ redirect to most recent snapshot matching that prefix `https://demo.archivebox.io/web/20230116145642/https://example.com/index.html`



Subtasks:
- [x] adds derived `ulid` field + migration to coalesce old uuid and timestamp fields into new ulid format (+asserts all snapshot timestamps are valid and are between 1900 and 2100 AD) (done in v0.8.5)
- [x] update admin and index UI to show ULID of old UUID4 `xxxx-xxxx-xxxxxxx` format, add ULID diagram in docs breaking it down into timestamp and randomness
- [x] create disambiguation page to show all the matching results for a given SLUG if it's the prefix for multiple possible snapshots
- [ ] reject Snapshot UUIDs being created that begin with `0`, `1`,`2`,`htt` to make prefix-matching faster and less error prone (avoids clashing with `199x*`/`20**` year, `1*` unix timestamp, `01*` ULIDs, or `http(s?)` URL slug prefixes)
- [ ] add docs examples on how to truly "self-host your own archive.org", add screenshot side-by-side of URL bar examples for visiting snapshots on Archive.org and demo.Archivebox.io

image

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the snapshot URL-routing code and existing timestamp or ULID lookup behavior, then trace the redirect and disambiguation paths. Done means the documented /web/ forms resolve or redirect correctly, including prefix matches and ambiguous results, with the remaining documentation and validation subtasks addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, web-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.