camelot-dev / camelot-dev/camelot

Camelot returns tables that contain no text (Where text should be detectable)

Open
#337 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
3.8k
Forks
546
Avg merge
3d 17h
Merged PRs (30d)
3

Description

I'm trying to extract data from some ~900 certificates. These certificates have an identical visual structure, but are published by different parties. For the majority of files the extraction works. However, for several dozen files, the table-structure returned by Camelot contains only empty strings.

Plotting `grid` and `text` shows content is detected (e.g. Table 7 in [DS_3663.pdf](https://github.com/camelot-dev/camelot/files/10046174/DS_3663.pdf)):
![DS_3663_table_7_grid](https://user-images.githubusercontent.com/36299/202836557-fd30d7c8-7be7-40fe-bcf1-c001f22a8833.png)

![DS_3663_table_7_text](https://user-images.githubusercontent.com/36299/202836590-f4c85ef9-381b-48d3-8dbd-48bbb5f8177c.png)

I'm using this command to read the pdf and create the tables:
`>>> tables=camelot.read_pdf('pdfs/DS_3663.pdf', pages='1-end', line_scale=110, shift_text=[''])`

e.g. Table 7 contains this data:
`>>> tables[7].data`
`[['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', ''], ['', '', '', '', '', '', '', '', '', '', '', '', '', '']]
`

Here are a few more example pdfs where the extraction fails in an identical manner:
[DS_885.pdf](https://github.com/camelot-dev/camelot/files/10046200/DS_885.pdf)
[DS_2481.pdf](https://github.com/camelot-dev/camelot/files/10046214/DS_2481.pdf)
[DS_2083.pdf](https://github.com/camelot-dev/camelot/files/10046204/DS_2083.pdf)

Parsing all of these files with `pdf2txt.py` successfully extracts text, so I assume it should be possible to get a result with Camelot as well.

**Environment**

- OS: Ubuntu 22.04.1 LTS
- Python version: 3.10.6
- Numpy version: 1.23.4
- OpenCV version: 4.6.0.66
- Ghostscript version: 9.55.0
- Camelot version: 0.9.0

I've tried debugging this, but had difficulties understanding the intricate code in the bbox-sections. From what I've figured out, it appears to me that Camelot is unable to marry `horizontal_text` (Which contain the relevant text) with the line-grid.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the empty-table result with camelot.read_pdf on DS_3663.pdf, especially Table 7, and compare it with DS_885.pdf, DS_2481.pdf, or DS_2083.pdf. Start by tracing how horizontal_text is associated with the line-grid in the bbox-sections code. Done means these PDFs return their detected text in tables[7].data rather than rows of empty strings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.