aws / aws/amazon-sagemaker-examples
LDA batch transform fails at scale (even on original training data)
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I have successfully trained an LDA model on a corpus of ~100k documents.
As a next step, I wish to predict the topic distribution associated with each document using batch transform. The document-term-matrix is in x_recordio-protobuf format.
My code is as follows:
```
transformer = lda.transformer(instance_count=1, instance_type='ml.c5.4xlarge', max_payload=1, output_path=batch_output, accept='application/x-recordio-protobuf')
transformer.transform(data=batch_input, data_type='S3Prefix', content_type='application/x-recordio-protobuf', split_type='RecordIO')
transformer.wait()
```
Using the original corpus as input results in an InternalServerError
Using a subset of the original corpus works so long as it is sufficiently small.
I have tried up to an ml.c4.18xlarge instance, as well as up to 5 instances of ml.c5.4xlarge. I have tried restricting max_payload and/or max_concurrent_transforms to 1. None of these help.
Here is the full error message:
```
ValueError: Error for Transform job lda-2019-06-04-22-56-24-198-2019-06-04-22-56-24-517: Failed Reason: InternalServerError: We encountered an internal error. Please try again.
```
Digging into logs:
the data log only contains the following:
```
MaxConcurrentTransforms=1, MaxPayloadInMB=1, BatchStrategy=MULTI_RECORD
temp-lda-results/lda-d100-predict.data: Unable to get response from algorithm
```
The main log contains, amongst many messages, 5 instances of the following two errors:
```
[CRITICAL] WORKER TIMEOUT (pid:89)
terminate called after throwing an instance of 'std::system_error'. what(): No such process
```
Again, it will run fine with a smaller set of documents, eg 2000 documents are processed without errors, but 20000 will throw this error.
UPDATE: I recently changed the LDA alpha0 parameter from 1 to 0.1. Now the error happens with as few as 500 documents being batch transformed.
Any assistance would be much appreciated.
Contributor guide
Research direction
Start by reproducing the shown LDA transformer.transform configuration with the 2,000- and 20,000-document inputs, then compare the data and main logs. Check the reported WORKER TIMEOUT and “Unable to get response from algorithm” messages, including the alpha0 update. Done means batch transform completes for the reported corpus sizes without the InternalServerError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100