Image tasks lose candidate labels and can't read chat responses on third-party providers
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### What happened?
When the Hugging Face Inference operator falls back from `hf-inference` to a third-party chat-completions provider, the image tasks degrade in two ways. This is the same bug class as #7195, in `ImageTaskCodegen` instead of `QaRankingCodegen`.
**1. Candidate labels are dropped from the request.** For `zero-shot-image-classification`, `ImageTaskCodegen.payloadPython` (L92-100) builds the native HF pipeline payload:
payload = {"inputs": , "parameters": {"candidate_labels": labels}}
Only `hf-inference` understands that shape. On a third-party provider, `_call_provider` rebuilds the request as an image part plus a text part (`HuggingFaceCodegenBase.scala` L369, L511, L532), where the text is just `prompt_value`. `parameters` is never carried over, so the provider receives an image with no candidate labels: the input that defines the task. The result is a generic caption instead of a classification, with no error reported.
`visual-question-answering` and `document-question-answering` are unaffected here, since their question is `prompt_value`, which the chat branch does send.
**2. Chat responses are written to the result column as raw JSON.** Chat providers reply with `{"choices": [{"message": {"content": ...}}]}`, and three image tasks cannot read that shape in `ImageTaskCodegen.parsePython`:
- `zero-shot-image-classification` (L147) is grouped with three other tasks and always returns `json.dumps(body)`.
- `visual-question-answering` and `document-question-answering` (L114) return `body.get("answer", json.dumps(body))`, and a chat response has no `answer` key.
So even when the provider answers correctly, the user sees the raw JSON envelope instead of the answer. `image-to-text` and `image-text-to-text` in the same file already handle `choices` correctly, so the fix is to apply that existing idiom to the remaining three tasks.
Both halves need fixing together: correcting only the request still displays raw JSON, and correcting only the response cleanly displays an answer to a question the model was never asked.
The same raw-JSON output appears for `visual-question-answering` and `document-question-answering` whenever they fall back to a chat provider.
### How to reproduce?
1. Add a Hugging Face Inference operator with task `zero-shot-image-classification`, an image column, and two or more Candidate Labels.
2. Choose a model whose cheapest provider is not `hf-inference`, or make `hf-inference` fail, so the request goes through `_call_provider`'s chat branch.
3. Run the workflow. The outbound request carries the image and the prompt text but no candidate labels, and the result column contains a raw chat JSON envelope rather than a label.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
Contributor guide
Assessment
This issue has not been assessed yet.