internetarchive / internetarchive/openlibrary

MARC dates need normalization to track changing cataloging practices

Open
#11,913 4 comments 0 reactions 0 assignees View on GitHub
Lead: @hornc Needs: Breakdown Needs: Response Priority: 3 Theme: MARC records Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

Some time over the past few years (or decade?) library cataloging practice has apparently switched from using "ca." to using "approximately" to designate uncertain dates. Similarly "fl." has morphed into "active" for `florit` / active dates.

You can see the effects of this with the duplicate author record that was created a couple of months ago because the dates weren't normalized, even though they match exactly.

[OL16119383A](https://openlibrary.org/authors/OL16119383A?v=1) Suetonius approximately 69 - approximately 122
[OL68512A](https://openlibrary.org/authors/OL68512A) Suetonius ca. 69 - ca. 122
You can also see that the Library of Congress has switched to the new standard:
https://id.loc.gov/authorities/names/n50053072.html

Term frequency counts for the 31 Jan 2026 dump:
```
14490 ca.
6506 approximately
161 c.

7410 cent.
1088 century

7222 active
2916 fl.
```

Although this affects relatively few authors, it tends to be authors of the classics and also has a knock-on effect in producing duplicate works.

### Reproducing the bug

1. Go to ...
2. Do ...

* Expected behavior: Date terms are normalized to a common representation which can be used for matching, avoiding false negative matches.
* Actual behavior:

### Context

- Browser (Chrome, Safari, Firefox, etc):
- OS (Windows, Mac, etc):
- Logged in (Y/N):
- Environment (prod, dev, local): prod

### Breakdown

Implementation Details (for maintainers)

#### Requirements Checklist
* [ ]

#### Related files

*

#### Stakeholders

* @hornc


#### Instructions for Contributors

- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Open Library parses and compares author date strings, using the duplicate Suetonius records and the listed MARC terms as examples. Identify the relevant implementation and tests, then define normalization behavior for equivalent terms such as “ca.” and “approximately” so matching no longer creates false duplicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.