huggingface / huggingface/transformers.js
Pix2Struct support
- Dominant language
- JavaScript
- Stars
- 16.3k
- Forks
- 1.2k
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 6
Description
### Model description
Pix2Struct is an image encoder - text decoder model that is trained on image-text pairs for various tasks, including image captionning and visual question answering.
https://huggingface.co/google/pix2struct-textcaps-base
https://huggingface.co/google/pix2struct-docvqa-base
https://huggingface.co/google/pix2struct-screen2words-base
### Prerequisites
- [X] The model is supported in Transformers (i.e., listed [here](https://huggingface.co/docs/transformers/index#supported-models-and-frameworks))
- [X] The model can be exported to ONNX with Optimum (i.e., listed [here](https://huggingface.co/docs/optimum/main/en/exporters/onnx/overview))
### Additional information
```Python
import requests
from PIL import Image
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)
model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-textcaps-base")
processor = Pix2StructProcessor.from_pretrained("google/pix2struct-textcaps-base")
# image only
inputs = processor(images=image, return_tensors="pt")
predictions = model.generate(**inputs)
print(processor.decode(predictions[0], skip_special_tokens=True))
>>> A stop sign is on a street corner.
```
This example is from the textcaps model page. Note that the processor also can take in text to ask questions about the image, like in the Docvqa version.
### Your contribution
I do not believe I have the knowledge to help make the interference logic.
Contributor guide
Assessment
This issue has not been assessed yet.