internetarchive / internetarchive/openlibrary
i18n pipeline last mile: local dev, olbase freshness, and .po file deprecation
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
## Context
We have been working to extract Open Library's i18n translation pipeline into a dedicated [`internetarchive/openlibrary-i18n`](https://github.com/internetarchive/openlibrary-i18n) repository. The core pipeline is now built and tested. This issue tracks the remaining integration questions before we can consider the migration complete.
Parent epic: #13061
---
## What's in flight
| PR | Description | Status |
|----|-------------|--------|
| [openlibrary-i18n #1](https://github.com/internetarchive/openlibrary-i18n/pull/1) | Translation pipeline infrastructure: `./i18n` toolbox, agent prompt, `translate.yml` workflow, `validate-pr.yml` CI gate, 50 tests | Ready to merge |
| [openlibrary #13069](https://github.com/internetarchive/openlibrary/pull/13069) | Dispatch trigger: fires `repository_dispatch` to `openlibrary-i18n` when `messages.pot` changes on master | Draft, depends on i18n #1 |
| [openlibrary #13070](https://github.com/internetarchive/openlibrary/pull/13070) | `Dockerfile.olbase`: `git clone openlibrary-i18n` at image build time to bake in latest translations | Draft, independent |
Merge order: **i18n #1 → #13069 → #13070** (though #13070 is independent of #13069).
---
## What works today
- `openlibrary-i18n` has 23 languages with `.po` files synced to the current `messages.pot`
- AI translation pipeline has been manually tested end-to-end (Romanian and Assamese PRs merged)
- PR validation CI (`validate-pr.yml`) runs format-string and HTML structure checks on every translation PR
- `openlibrary-bot` is confirmed able to open PRs and issues on both repos
---
## Last mile questions
### 1. Local dev translations after `.po` removal
The current plan (PR #13070) bakes translations into `olbase` at image build time. But local dev uses `OL_MOUNT_DIR` which volume-mounts `openlibrary/openlibrary/i18n/` into the container — **shadowing** whatever olbase baked in. So removing the committed `.po` files from this repo would leave local dev with no translations.
**Options:**
- `make i18n-sync` target that clones/pulls from `openlibrary-i18n` locally on demand
- Keep `.po` files in this repo as a "cache" (updated by a bot PR when `openlibrary-i18n` changes) — two sources of truth but local dev still works
- Accept that local dev runs without translations until someone runs a sync command — document it clearly in CONTRIBUTING.md
### 2. olbase rebuild frequency
olbase rebuilds weekly (Tuesdays) + on `workflow_dispatch`. After a translation PR merges to `openlibrary-i18n`, it takes up to a week to land in a deployed container.
**Is that acceptable?** If not, options:
- Trigger an olbase rebuild via `workflow_dispatch` after each `openlibrary-i18n` merge (adds a dispatch step to `openlibrary-i18n`'s merge flow)
- Move from baking translations into olbase to mounting them at runtime from a separate image or volume
### 3. Deprecation path for `.po` files in this repo
The `.po` files in `openlibrary/openlibrary/i18n/` are the authoritative source for deployed translations today. After #13070 merges, olbase gets translations from `openlibrary-i18n` instead — but only for production builds. The files in this repo become stale over time.
**Questions:**
- Do we want a bot PR that periodically syncs `openlibrary-i18n` translations back into this repo (keeping both in sync)?
- Or do we deprecate/remove them, accepting a migration cost for local dev?
- The `scripts/i18n-messages` script (`update`, `compile`, `validate`) still works against the local `.po` files — does that toolchain need to change?
### 4. `messages.pot` trigger reliability
PR #13069 fires when `messages.pot` changes on master. The `generate-pot` pre-commit hook regenerates `messages.pot` on every PR — but it only updates the file if source strings actually changed. If no new strings are added for months, no dispatch fires.
**Should we add a weekly/monthly fallback schedule** to `translate.yml` in `openlibrary-i18n` (independent of the dispatch) to catch any drift? Currently the workflow only triggers on `repository_dispatch`.
### 5. Contributor and documentation updates
Once the pipeline is live:
- CONTRIBUTING.md should note that translation is now handled automatically via `openlibrary-i18n`
- `openlibrary/openlibrary/i18n/README.md` (if it exists) should point to the new repo
- The `Makefile` targets `make i18n` and `make test-i18n` still reference local `.po` files — do they need updating?
---
## Suggested next steps
1. Merge openlibrary-i18n #1 (no blockers)
2. Trigger the full pipeline via `workflow_dispatch` in `openlibrary-i18n` to validate all 23 languages end-to-end in CI
3. Merge #13069 (dispatch trigger) — low risk, no user-visible effect
4. Decide on local dev strategy (question 1 above) before merging #13070
5. File follow-up issues for olbase rebuild frequency and `.po` deprecation once #13070 is merged and stable
Contributor guide
Research direction
Start with Dockerfile.olbase, OL_MOUNT_DIR, scripts/i18n-messages, the Makefile i18n targets, and the referenced translate.yml workflow. Trace local and production translation paths, then document the chosen local-dev, rebuild, deprecation, and fallback strategies; done means the migration decisions and contributor updates are explicit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, python
- Domain
- build-system, devops, documentation, internationalization
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100