Question about gpt2 model

Ask a Question

How should be the input when we want to do prediction on batch_size more than 1?

Question

Explain your question here.
this one:
texts = ["Here is some text to encode : Hello World!", "Here is some text to encode : Hello World!"]
input_ids_1 = [[tokenizer.encode(text, add_special_tokens=True) for text in texts]]

or this one:
texts = ["Here is some text to encode : Hello World!", "Here is some text to encode : Hello World!"]
input_ids_1 = [[tokenizer.encode(text, add_special_tokens=True)] for text in texts]

The get_inputs() API for onnx model returns: ['input1_dynamic_axes_1', 'input1_dynamic_axes_2', 'input1_dynamic_axes_3'] as input shape but it is not obvious which one corresponds to batch size and which one to sequence length? What is the third dimension?

Further information

Relevant Area (e.g. model usage, backend, best practices, pre-/post- processing, converters):

Is this issue related to a specific model? gpt2 model https://github.com/onnx/models/blob/main/text/machine_comprehension/gpt-2/model/gpt2-10.onnx
Model name (e.g. mnist):
Model opset (e.g. 7):

Notes

Any additional information, code snippets.

Start with the referenced gpt2-10.onnx model and inspect the get_inputs() shapes alongside the tokenizer examples in the issue. Determine how the three dynamic dimensions represent batching, sequence length, and the remaining input, then document the valid batched input form and what each dimension means.