docling-project / docling-project/docling
Pdf2parquet improvement and optimisation
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 95
Description
### Requested feature
Hello Docling team,
I have been working around the data prep kit repository for sometime now and I found out one of the major issue over there with the speed of pdf2parquet module. There was a high priority issue open over there for the speed of pdf2parquet which is based on docling . After reading through a bit more I came to know few primary causes why this module is soo slow and consumes soo much computational power. I pointed out according to my understanding the issue in **pdf backend and the always on ocr for the slow speed**. I mentioned a few changes in the data prep kit but as it is based on docling ,@dolfim-ibm guided me to open a proper issue here with proper report so that it can be solved for everyone upfront.
So before I opened this issue here on the repo I have been throughly testing the code for a few days to make sure the test reports are accurate. Although in this phase I have tested mostly on just one pdf which is the smoldocling research paper only which is published.
_So here are my findings_:
I found out the one of the major culprits for the slower speed is pdf backend at the moment docling uses ```DLPARSE_V2 ``` as the default backend but also supports ```DLPARSE_V4``` and ```PYPDFIUM 2```
I tested out the code on all 3 backends multiple times for a lot of random test runs and here are my findings :
1.```DLPARSE_V4``` and ```DLPARSE_V2 ``` are neck in neck for the timing I did tests in multiple batches for the same file so I have noticed for each run the difference in the time can vary for about ±10% on an average. SO if you avearge out the number of runs and their run time ```DLPARSE_V4``` and ```DLPARSE_V2 ``` will have a difference of just ±1% which is interesting because that can allow us to shift our pdf backend to ```DLPARSE_V4``` whose support already exists , but we still dont use it actually for our backend I am not sure why . The time tests prove that the time wont change if we switch from ```DLPARSE_V2 ``` to ```DLPARSE_V4```, but we would get a better version of pdf backend which can process even better layouts as compared to ```DLPARSE_V2 ``` like forms ,receipts , etc .Therefore I think we should shift our default backend to ```DLPARSE_V4``` .
2.Another special backend is ```PYPDFIUM2``` because as much as I know its completely written in C++ so technically is fastest among all 3 . To be precise it is almost 15% to 20% faster. The only problem is , It **does not** support complex layouts and structures like tables while converting to parquet. Although I cannot recommend to use this by default due to just raw text extraction and other limitations but I have used ```PYPDFIUM2``` earlier a lot and I think it does a decent job in processing the pdfs especially which does not contain tables or complex layouts like IEEE papers. Most of the commerical users I think maybe would not be directly dealing with such complex structures/ layouts I think most of them have normal pdfs. I was thinking can we use ```PYPDFIUM2``` as a special param maybe where if say user states that "no_complex_pdfs" param is True it will shift to ```PYPDFIUM2``` as backend. This param pipeline option is off say by default . And by default the pdf backend can be ```DLPARSE_V4``` if we can shift after point 1 possibly .
3. The ocr is on by defualt I think with easyocr and is on by default. I think this is essential for the accuracy of ingestion docling is famous for I think . However , I would like to test these backends without ocr too.
Also I need some help in comparing the parquets , Although I experimented with computational power and the runtime for conversion . I could not compare these parquets.. if the team guide me a bit about how to compare these parquets too maybe I can give you a second verison of this report in which I can go in detail for all of the above points. @dolfim-ibm
The only way I till have found to compare these huge files is possibly load the parquet context in two strings and compare it using some sentence transformer to get a context similarity score . Although I think these backends are pretty mature soo similarity scores are going to be high anyways.
I would love to discuss this further with the team and would absolutely love if I can submit a PR for the same !
Contributor guide
Research direction
Start by reproducing the reported pdf2parquet timing comparisons across DLPARSE_V2, DLPARSE_V4, and PYPDFIUM2, with OCR enabled and disabled. Compare the resulting parquet content and document the supported layout trade-offs. Done should include an agreed backend and OCR behavior, evidence across representative PDFs, and regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100