consider a semi-automated, crowd-sourced approach

I'm doing research on annotation and sensemaking with research articles (with CMU/UMD) and I needed a json object of section hierarchies and paragraphs that's close to perfect. I typically extract the figures with deepfigs2. I'm using https://github.com/coolwanglu/pdf2htmlEX to create an almost perfect copy of the pdf as html and then using some jquery to click out the section hierarchies and paragraphs starts/ends, which looks like this: https://media.giphy.com/media/fxgTDf3defTd8eSbK2/giphy.gif. Each line is a div which maintains reading order very well, which means an RNN/CNN could probably figure out what most lines are based on the the div dimensions, text content, and the distances between divs. Having a UI such as the above could give readers a way to fix the last 1-40% of errors I've seen in every pdf conversion, and create more training data with each fixed conversion. This should lead to less and less manual fixing, but I'd bet that nothing will ever be perfect for every pdf and readers will need a way to visually fix errors. Maybe some form of this semi-automated approach could valuable to readers on semantic scholar that would want their library of papers to be machine readable, responsive (like the new arxiv feature), and therefore have the most cutting edge reading/searching experience, all while ending the tyranny of the pdf.