google-research / google-research/pix2struct

The pre-trained checkpoint generates very short output

Open
#38 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
690
Forks
61
PR merge metrics
No merged PRs in 30d

Description

Thanks for your awesome work!

I want to utilize the model to generate the HTML of an image, so I choose the pre-trained checkpoint without fine-tuning. However, the generated output is very short. For example, the following code only generate `` without any detailed struct.

```
from PIL import Image
import torch
from transformers import Pix2StructProcessor, Pix2StructForConditionalGeneration
device = torch.device("cuda")
processor = Pix2StructProcessor.from_pretrained("google/pix2struct-large")
model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-large").to(device)

img_path = 'biography.png'
image = Image.open(img_path)
processor.image_processor.is_vqa=False

inputs = processor(images=image, return_tensors="pt").to(device)
generated_ids = model.generate(**inputs, max_length=1000)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_text)
```
My transformers version is 4.28.0. Do you know how to solve this problem? Thanks in advance :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.