docling-project / docling-project/docling-core
feat: Allow to `export_to_dataframe` to different kind of dataframes
- Dominant language
- HTML
- Stars
- 282
- Forks
- 214
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 23
Description
# Description
Would you consider adding a parameter to `export_to_dataframe` to enable exporting to pandas, polars or pyarrow?
## Motivation
In recent times, I find myself avoiding installing pandas whenever I can if I can get away with polars only (there are quite a few reasons for this)
`docling-core` requires pandas as a strict dependency only to enable the `export_to_dataframe` feature. Making that optional would lower the dependency burden and all the transitive dependencies that come with that.
## Proposal
I am one of the maintainer of [Narwhals](https://github.com/narwhals-dev/narwhals) (An extremely lightweight and extensible compatibility layer between dataframe libraries) and I would be happy to submit a PR to enable exporting to different dataframe libraries. Here is the [branch/changes](https://github.com/docling-project/docling-core/compare/main...FBruzzesi:docling-core:feat/allow-to-export-different-frame-type)
Remark that Narwhals comes dependency free, which means that has no real impact in the dependency tree, and it's up to the user to have installed the library to which they would like to export to.
Narwhals is used with the very same scope by libraries such as altair, plotly, bokeh, marimo and many others (see [ecosystem](https://narwhals-dev.github.io/narwhals/ecosystem/#used-by) to know more).
---
Concretely, the changes would look like something like the following (for a full diff, you can check the [branch/changes](https://github.com/docling-project/docling-core/compare/main...FBruzzesi:docling-core:feat/allow-to-export-different-frame-type) on my fork):
```diff
+ import narwhals.stable.v2 as nw
def export_to_dataframe(
self,
doc: Optional["DoclingDocument"] = None,
+ return_type: Literal["pandas", "polars", "pyarrow"] = "pandas"
):
...
+ df = nw.from_dict(data, backend=return_type) # <- this is a narwhals DataFrame, backed by either pandas, polars or pyarrow
+ return df.to_native() # <- this is the native dataframe
```
### Guarantees
We try to make two guarantees for projects:
1. Stable versions of the library, or [Perfect backwards compatibility policy](https://narwhals-dev.github.io/narwhals/backcompat/), which TL;DR is: we (almost) never ever do breaking changes on stable versions.
2. We test in our CI downstream dependencies that use narwhals (see [downstream_tests.yml](https://github.com/narwhals-dev/narwhals/blob/main/.github/workflows/downstream_tests.yml)).
## Related issues
https://github.com/docling-project/docling/issues/498
Contributor guide
Research direction
Start at the export_to_dataframe entry point and compare the proposed changes in the linked branch. Check how the current pandas export and dependency handling work, then verify that the requested pandas, polars, and pyarrow return types preserve existing behavior and are covered by tests or CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100