docling-project / docling-project/docling

DocumentStream only accepts a BytesIO, but it should instead accept an abstract class like` IO[bytes]`

Open
#2,279 2 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature

It appears that you can only construct a `DocumentStream` from a `BytesIO` concrete class. And this is great for most general purpose use cases, but if you happen to have a custom type that implements the `IO` abstract class methods such as `read`, `seek`, etc, then `DocumentStream` should allow this as well.

A concrete use case for this is [Daft](https://github.com/Eventual-Inc/Daft) has it's own `daft.File` class that implements the abstract methods of `read`, `seek`, etc. But there is no way to use this as a source in docling.

A few concrete examples of why this flexibility is important.

### Supporting integrations with other libraries like [Daft](https://github.com/Eventual-Inc/Daft)

```py
import daft
from docling.document_converter import DocumentConverter

file = daft.File("s3://my_bucket/my_doc.pdf")

converter = DocumentConverter()
result = converter.convert(file) # this won't work because of the type constraints on `DocumentStream`
```

### Better integration with other types such as `FileIO`
```py
from io import FileIO

file_io = FileIO(filepath, 'rb')
converter = DocumentConverter()
result = converter.convert(file) # this also won't work because of the type constraints on `DocumentStream`
```

### Alternatives
The only alternative is to materialize the document by converting it to bytesio. But this is extremely inefficient.

```py
import daft
from docling.document_converter import DocumentConverter

file = daft.File("s3://my_bucket/my_doc.pdf")
bytes = file.read()
bytesio = BytesIO(bytes)

converter = DocumentConverter()
ds = DocumentStream(name="test", stream=bytesio)
result = converter.convert(ds)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.