Document Location Mapping

Document Location Mapping

The main goal of document location mapping is to match segments (token ranges) with structural elements in the original documents (pages and poetic lines). This will allow various tools (e.g rolling window) to display the original page and line numbers instead of meaningless token ranges or segment numbers. Currently, we can only do this if we cut by milestone. We'll want to expand this functionality to transform xml tags into milestones. That way a TEI text with chapter divisions encoded can be used.

Ideally, we'd have some sort of index that keeps track of the token ranges for the divisions in the current session. The main challenge is that, once a document is scrubbed and/or tokenised, its feature vector is not the same as the original. Sometimes the token count can be significantly reduced (or occasionally increased). So mapping token ranges onto document strings becomes a sequence alignment problem.

Start by tracing the existing milestone-based location handling and the scrubbing and tokenisation flow. The issue calls for an index mapping token ranges to original pages and poetic lines, including TEI divisions, while accounting for changed token counts. Done means structural locations can be recovered for transformed documents without relying only on milestones.