apache / apache/texera

Image tasks lose candidate labels and can't read chat responses on third-party providers

Open
#7,906 1 comment 0 reactions 1 assignee Claimed by @PG1204 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### What happened?

When the Hugging Face Inference operator falls back from `hf-inference` to a third-party chat-completions provider, the image tasks degrade in two ways. This is the same bug class as #7195, in `ImageTaskCodegen` instead of `QaRankingCodegen`.

**1. Candidate labels are dropped from the request.** For `zero-shot-image-classification`, `ImageTaskCodegen.payloadPython` (L92-100) builds the native HF pipeline payload:

payload = {"inputs": , "parameters": {"candidate_labels": labels}}

Only `hf-inference` understands that shape. On a third-party provider, `_call_provider` rebuilds the request as an image part plus a text part (`HuggingFaceCodegenBase.scala` L369, L511, L532), where the text is just `prompt_value`. `parameters` is never carried over, so the provider receives an image with no candidate labels: the input that defines the task. The result is a generic caption instead of a classification, with no error reported.

`visual-question-answering` and `document-question-answering` are unaffected here, since their question is `prompt_value`, which the chat branch does send.

**2. Chat responses are written to the result column as raw JSON.** Chat providers reply with `{"choices": [{"message": {"content": ...}}]}`, and three image tasks cannot read that shape in `ImageTaskCodegen.parsePython`:

- `zero-shot-image-classification` (L147) is grouped with three other tasks and always returns `json.dumps(body)`.
- `visual-question-answering` and `document-question-answering` (L114) return `body.get("answer", json.dumps(body))`, and a chat response has no `answer` key.

So even when the provider answers correctly, the user sees the raw JSON envelope instead of the answer. `image-to-text` and `image-text-to-text` in the same file already handle `choices` correctly, so the fix is to apply that existing idiom to the remaining three tasks.

Both halves need fixing together: correcting only the request still displays raw JSON, and correcting only the response cleanly displays an answer to a question the model was never asked.

The same raw-JSON output appears for `visual-question-answering` and `document-question-answering` whenever they fall back to a chat provider.

### How to reproduce?

1. Add a Hugging Face Inference operator with task `zero-shot-image-classification`, an image column, and two or more Candidate Labels.
2. Choose a model whose cheapest provider is not `hf-inference`, or make `hf-inference` fail, so the request goes through `_call_provider`'s chat branch.
3. Run the workflow. The outbound request carries the image and the prompt text but no candidate labels, and the result column contains a raw chat JSON envelope rather than a label.

### Version/Branch

1.3.0-incubating-SNAPSHOT (main)

### Commit Hash (Optional)

_No response_

### What browsers are you seeing the problem on?

_No response_

### Relevant log output

```shell

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.