Batch predictions to run on GPU

It is common when deploying ML models to queue up requests to run them on the GPU all at once to increase throughput.

This is distinct from users being able to run several predictions in one go with the API. We are calling this a "bulk" API to disambiguate. GPU batching is purely to make running predictions more efficient on a GPU, and should be independent of how many predictions a user submits. The model should determine how many predictions is needs to pull of the queue to maximize throughput.

Requirements

• Allow the user to define a predict function that can take multiple inputs. A batch prediction is a superset of a single prediction, so we may as well just allow one or the other. It means you get better performance, at the expense of it being harder to implement. "If you want to use batching, adapt your predict function to take a list of inputs."
• Queue up HTTP/Redis requests. There is a trade-off between latency and throughput. Most serving systems seem to let the user define a max batch size and a max latency/timeout. But, that will mean predictions will always take at least the timeout unless the queues are saturated. Maybe there's something more clever we can do there?

Potential design

We would probably need to define the input type out of the function signature, unlike a single prediction.

class InputObj(BaseModel):
    image: File
    another_param: float = Input(default=0.26)

class Predictor(BasePredictor):
    def predict(self, inputs: List[InputObj] = Batch(batch_size=25, timeout=200)):
        processed_inputs = map(inputs, preprocess)
        return self.model(processed_inputs)

Maybe batch size and timeout should be configurable at runtime, and these are considered defaults?

Future work

• Pre-/post-processing is often done on CPU so could be done separately.

Prior art

• https://github.com/tensorflow/serving/blob/master/tensorflow_serving/batching/README.md
• https://pytorch.org/serve/batch_inference_with_ts.html

User data

• @daanelson says this would make language models much faster.

/cc @DeNeutoy

No repository file, test, or entry point is named; begin by locating the predictor API and HTTP/Redis request handling, then compare the TensorFlow Serving batching README and PyTorch Serve batch-inference example. Done means supporting predictor functions that accept batches and queuing requests with configurable batch-size and latency behavior.