Torchscript scripting C++ model for batch inference

Hi,

As known from the detectron2 deployment description, the detectron2 TorchScript scripting model supports dynamic batch_size. I am currently working on modifying the official example "[torchscript_mask_rcnn.cpp]" into batch inference with batch_size>1. However, it does not works.

//create a Tuple[Dict[str, Tensor]] which is the input type of scripted model

c10::IValue get_scripting_inputs(cv::Mat& img, c10::Device device) {
const int height = img.rows;
const int width = img.cols;
const int channels = 3;

auto img_tensor =
torch::from_blob(img.data, { height, width, channels }, torch::kUInt8);
// HWC to CHW
img_tensor =
img_tensor.to(device, torch::kFloat).permute({ 2, 0, 1 }).contiguous();

cout << "img_tensor" << img_tensor.sizes() << endl;
auto img_tensor_l = img_tensor.unsqueeze(0);
cout << "img_tensor_l" << img_tensor_l.sizes() << endl;

auto dic = c10::Dict();
dic.insert("image", img_tensor);
return std::make_tuple(dic);
}

1, The example mentions "create a Tuple[Dict[str, Tensor]] which is the input type of scripted model", so I have tried to create a

Tuple[Dict[str, Tensor], Dict[str, Tensor]]

by using

return std::make_tuple(dic, dic)

in the return of the above function get_scripting_inputs. However, it reports the errors

"Expected a value of type 'Tuple[Dict[str, Tensor]]' for argument 'inputs' but instead found type 'Tuple[Dict[str, Tensor], Dict[str, Tensor]]'."

2, Alternatively, I have tried to stack the inputs as:

auto inputs = get_scripting_inputs(input_img_resize, device);
std::vector inputs_list;

for (int i = 0; i < 5; i++) {
inputs_list.push_back(inputs);
}
c10::Stack stack{ inputs_list };
auto outputs = model.forward({ stack });

However, it gives the error "Expected at most 2 argument(s) for operator 'forward', but received 6 argument(s). Declaration: forward(__torch__.ScriptableAdapter self, (Dict(str, Tensor)) inputs) -> (Dict(str, Tensor)[])".

Can anyone provide some advices on how to realize the batch inference on the Torchscript scripting? Many Thanks.