googleapis / googleapis/python-aiplatform

Batch prediction results are not guaranteed to be in the right order

Đang mở
#1,166 5 bình luận 1 reaction 1 người được giao Được giao cho @weichungw Xem trên GitHub
aiplatform api: vertex-ai type: feature request
Ngôn ngữ chính
Python
Star
905
Fork
465
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
44

Mô tả

#### Environment details

- OS type and version: MacOS 12.3.1
- Python version: 3.9.10
- pip version: 22.0.3
- `google-cloud-aiplatform` version: 1.12.0

#### Steps to reproduce

1. run a batch `BatchPredictionJob` that outputs multiple `prediction-results-xxxxx-to-xxxxx` files.
2. the order of the predictions does not always match the order of the inputs.

#### Code example

This happens with official code examples such as [sdk-custom-image-classification-batch.ipynb](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/36455b8125802459f3a40752fbda0e4c9407c854/notebooks/official/custom/sdk-custom-image-classification-batch.ipynb). The relevant part of the code is this:

```python
# Get downloaded results in directory
results_files = []
for dirpath, subdirs, files in os.walk(latest_directory):
for file in files:
if file.startswith("prediction.results"):
results_files.append(os.path.join(dirpath, file))

# Consolidate all the results into a list
results = []
for results_file in results_files:
# Download each result
with open(results_file, "r") as file:
results.extend([json.loads(line) for line in file.readlines()])

```

Firstly, `os.walk()` does not guarantee the order. In practice, it seems to respect the order, but it's brittle to count on this.

Secondly, and more importantly, I've run into cases where the files were not in the same order as the inputs. I would get 7% accuracy on MNIST, then by just reversing the order of the prediction files, I would get 100%.

Thirdly, I haven't tested it but I suspect that the order would also be wrong if there's any error on any instance.

Lastly, the inputs may sometimes be large, and it's not efficient to include them in the predictions. I would much rather have an input identifier, such as its source file and its line index.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.