[Bug]: 16177 has type numpy.int64, but expected one of: bytes
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
When running TFX's Evaluator component (which uses `tensorflow_model_analysis` to create beam jobs), the following error occurs in one of the workers, while all the other workers was able to complete successfully:
Traceback:
```
Error message from worker: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 287, in _execute
response = task()
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 360, in
lambda: self.create_worker().do_instruction(request), request)
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 596, in do_instruction
return getattr(self, request_type)(
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
monitoring_infos = bundle_processor.monitoring_infos()
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1139, in monitoring_infos
op.monitoring_infos(transform_id, dict(tag_to_pcollection_id)))
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/operations.py", line 543, in monitoring_infos
all_monitoring_infos.update(self.user_monitoring_infos(transform_id))
File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/operations.py", line 584, in user_monitoring_infos
return self.metrics_container.to_runner_api_monitoring_infos(transform_id)
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/execution.py", line 309, in to_runner_api_monitoring_infos
all_metrics = [
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/execution.py", line 310, in
cell.to_runner_api_monitoring_info(key.metric_name, transform_id)
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/cells.py", line 76, in to_runner_api_monitoring_info
mi = self.to_runner_api_monitoring_info_impl(name, transform_id)
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/cells.py", line 150, in to_runner_api_monitoring_info_impl
return monitoring_infos.int64_user_counter(
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/monitoring_infos.py", line 185, in int64_user_counter
return create_monitoring_info(
File "/usr/local/lib/python3.9/site-packages/apache_beam/metrics/monitoring_infos.py", line 302, in create_monitoring_info
return metrics_pb2.MonitoringInfo(
TypeError: 7006 has type numpy.int64, but expected one of: bytes
```
This error also happens only occasionally, but frequent enough to break production pipeline with a recurring schedule. I would like to understand the root cause of this error to prevent issues in production.
Related issue in TFMA: https://github.com/tensorflow/model-analysis/issues/171
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start with apache_beam/metrics/monitoring_infos.py at create_monitoring_info and trace the value from apache_beam/metrics/cells.py through int64_user_counter; the traceback identifies these entry points. Reproduce the intermittent TFX Evaluator/Dataflow worker failure and determine why numpy.int64 reaches the MonitoringInfo protobuf. Done means the cause is documented and the worker no longer fails on this metric value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- cloud, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100