internetarchive / internetarchive/openlibrary

Add Table of Contents text from IA to book pages

Open
#3,237 8 comments 1 reaction 1 assignee Claimed by @cdrini View on GitHub
Affects: Experience Lead: @cdrini Priority: 3 Theme: Distribution Type: Epic Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

When searching for a book and landing on the books page, the books page does not the table of content of the book in `About the Book` even if the book has a table of contents. eg https://openlibrary.org/books/OL10295090M/Search_Engine_Optimization

Therefore there is an opportunity here to improve our website and content by Adding Table of Contents(we have ~500k) text for every book where we already have it into OpenLibrary.org book pages (from archive.org items) using [BookReaderGetTextWrapper.php](https://ia600607.us.archive.org/BookReader/BookReaderGetTextWrapper.php?path=%2F6%2Fitems%2Fsearchengineopti00kris%2Fsearchengineopti00kris_djvu.xml&page=5) & scandata.xml

The various forms of IA data are available from https://archive.org/download/searchengineopti00kris and the TOC can be derived from https://archive.org/download/searchengineopti00kris/searchengineopti00kris_scandata.xml
(although there's also a likely looking https://archive.org/download/searchengineopti00kris/searchengineopti00kris_toc.xml which is protected, perhaps because this is a copyrighted book).

The _scandata.xml file has TOC pages tagged with Contents and that information can be used to extract the appropriate text from the searchengineopti00kris_djvu.xml text file, or, in the case of protected books such as this one, fetched through the closed source API that the bookreader uses to get its Read Aloud text.

One significant complication is that converting raw page text into structured TOC data, in the syntax used by Open Library, is a non-trivial task. Using the raw text and letting the humans do the parsing cerebrally would be much easier.

### Stakeholders

@mekarpeles @tabshaikh

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.