internetarchive / internetarchive/openlibrary
Add Table of Contents text from IA to book pages
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
When searching for a book and landing on the books page, the books page does not the table of content of the book in `About the Book` even if the book has a table of contents. eg https://openlibrary.org/books/OL10295090M/Search_Engine_Optimization
Therefore there is an opportunity here to improve our website and content by Adding Table of Contents(we have ~500k) text for every book where we already have it into OpenLibrary.org book pages (from archive.org items) using [BookReaderGetTextWrapper.php](https://ia600607.us.archive.org/BookReader/BookReaderGetTextWrapper.php?path=%2F6%2Fitems%2Fsearchengineopti00kris%2Fsearchengineopti00kris_djvu.xml&page=5) & scandata.xml
The various forms of IA data are available from https://archive.org/download/searchengineopti00kris and the TOC can be derived from https://archive.org/download/searchengineopti00kris/searchengineopti00kris_scandata.xml
(although there's also a likely looking https://archive.org/download/searchengineopti00kris/searchengineopti00kris_toc.xml which is protected, perhaps because this is a copyrighted book).
The _scandata.xml file has TOC pages tagged with Contents and that information can be used to extract the appropriate text from the searchengineopti00kris_djvu.xml text file, or, in the case of protected books such as this one, fetched through the closed source API that the bookreader uses to get its Read Aloud text.
One significant complication is that converting raw page text into structured TOC data, in the syntax used by Open Library, is a non-trivial task. Using the raw text and letting the humans do the parsing cerebrally would be much easier.
### Stakeholders
@mekarpeles @tabshaikh
Contributor guide
Assessment
This issue has not been assessed yet.