docling-project / docling-project/docling-core

feat: Allow to `export_to_dataframe` to different kind of dataframes

Open
#382 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
282
Forks
214
Avg merge
2d 16h
Merged PRs (30d)
23

Description

# Description

Would you consider adding a parameter to `export_to_dataframe` to enable exporting to pandas, polars or pyarrow?

## Motivation

In recent times, I find myself avoiding installing pandas whenever I can if I can get away with polars only (there are quite a few reasons for this)

`docling-core` requires pandas as a strict dependency only to enable the `export_to_dataframe` feature. Making that optional would lower the dependency burden and all the transitive dependencies that come with that.

## Proposal

I am one of the maintainer of [Narwhals](https://github.com/narwhals-dev/narwhals) (An extremely lightweight and extensible compatibility layer between dataframe libraries) and I would be happy to submit a PR to enable exporting to different dataframe libraries. Here is the [branch/changes](https://github.com/docling-project/docling-core/compare/main...FBruzzesi:docling-core:feat/allow-to-export-different-frame-type)

Remark that Narwhals comes dependency free, which means that has no real impact in the dependency tree, and it's up to the user to have installed the library to which they would like to export to.

Narwhals is used with the very same scope by libraries such as altair, plotly, bokeh, marimo and many others (see [ecosystem](https://narwhals-dev.github.io/narwhals/ecosystem/#used-by) to know more).

---

Concretely, the changes would look like something like the following (for a full diff, you can check the [branch/changes](https://github.com/docling-project/docling-core/compare/main...FBruzzesi:docling-core:feat/allow-to-export-different-frame-type) on my fork):

```diff
+ import narwhals.stable.v2 as nw

def export_to_dataframe(
self,
doc: Optional["DoclingDocument"] = None,
+ return_type: Literal["pandas", "polars", "pyarrow"] = "pandas"
):
...
+ df = nw.from_dict(data, backend=return_type) # <- this is a narwhals DataFrame, backed by either pandas, polars or pyarrow
+ return df.to_native() # <- this is the native dataframe
```

### Guarantees

We try to make two guarantees for projects:

1. Stable versions of the library, or [Perfect backwards compatibility policy](https://narwhals-dev.github.io/narwhals/backcompat/), which TL;DR is: we (almost) never ever do breaking changes on stable versions.
2. We test in our CI downstream dependencies that use narwhals (see [downstream_tests.yml](https://github.com/narwhals-dev/narwhals/blob/main/.github/workflows/downstream_tests.yml)).

## Related issues

https://github.com/docling-project/docling/issues/498

Contributor guide

Open the contributing guide

Research direction

Start at the export_to_dataframe entry point and compare the proposed changes in the linked branch. Check how the current pandas export and dependency handling work, then verify that the requested pandas, polars, and pyarrow return types preserve existing behavior and are covered by tests or CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.