[QST] TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

❓ Questions & Help

Details

InferenceServerException Traceback (most recent call last)
Cell In[48], line 22
19 outputs = ["ordered_ids", "ordered_scores"]
21 # 4. Send request to Triton
---> 22 response = send_triton_request(request_schema, request, outputs)
24 # 5. Inspect the response
25 print(response)

File D:\build-rec\building-rec.venv\lib\site-packages\merlin\systems\triton\utils.py:230, in send_triton_request(schema, inputs, outputs_list, client, endpoint, request_id, triton_model)
226 triton_inputs = triton.convert_df_to_triton_input(schema, inputs, grpcclient.InferInput)
228 outputs = [grpcclient.InferRequestedOutput(col) for col in outputs_list]
--> 230 response = client.infer(triton_model, triton_inputs, request_id=request_id, outputs=outputs)
232 results = {}
233 for col in outputs_list:

File D:\build-rec\building-rec.venv\lib\site-packages\tritonclient\grpc_client.py:1572, in InferenceServerClient.infer(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm, parameters)
1570 return result
1571 except grpc.RpcError as rpc_error:
-> 1572 raise_error_grpc(rpc_error)

File D:\build-rec\building-rec.venv\lib\site-packages\tritonclient\grpc_utils.py:77, in raise_error_grpc(rpc_error)
65 def raise_error_grpc(rpc_error):
66 """Raise an :py:class:InferenceServerException from a gRPC error.
67
68 Parameters
(...)
75 InferenceServerException
76 """
---> 77 raise get_error_grpc(rpc_error) from None

InferenceServerException: [StatusCode.INTERNAL] Traceback (most recent call last):
File "/models/executor_model/1/model.py", line 97, in execute
File "/usr/local/lib/python3.10/dist-packages/merlin/systems/dag/ensemble.py", line 78, in transform
return runtime.transform(self.graph, transformable)
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/runtime.py", line 53, in transform
return self.executor.transform(transformable, [graph.output_node])
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 102, in transform
transformed_data = self._execute_node(node, transformable, capture_dtypes, strict)
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 116, in _execute_node
upstream_outputs = self._run_upstream_transforms(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 130, in _run_upstream_transforms
node_output = self._execute_node(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 116, in _execute_node
upstream_outputs = self._run_upstream_transforms(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 130, in _run_upstream_transforms
node_output = self._execute_node(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 116, in _execute_node
upstream_outputs = self._run_upstream_transforms(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 130, in _run_upstream_transforms
node_output = self._execute_node(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 116, in _execute_node
upstream_outputs = self._run_upstream_transforms(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 130, in _run_upstream_transforms
node_output = self._execute_node(
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 122, in _execute_node
transform_output = self._run_node_transform(node, transform_input, capture_dtypes, strict)
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 250, in _run_node_transform
raise exc
File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 237, in _run_node_transform
transformed_data = node.op.transform(selection, input_data)
File "/usr/local/lib/python3.10/dist-packages/merlin/systems/dag/ops/feast.py", line 241, in transform
feature_array = array_constructor(feature_value).astype(
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

Start with merlin/systems/dag/ops/feast.py around transform line 241, then trace the call from send_triton_request through the Triton executor shown in the traceback. Reproduce the failing inference and identify which feature value is None. Done means the failure path has defined, tested behavior for that input.