aws / aws/amazon-sagemaker-examples

LDA batch transform fails at scale (even on original training data)

Open
#755 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I have successfully trained an LDA model on a corpus of ~100k documents.
As a next step, I wish to predict the topic distribution associated with each document using batch transform. The document-term-matrix is in x_recordio-protobuf format.

My code is as follows:

```
transformer = lda.transformer(instance_count=1, instance_type='ml.c5.4xlarge', max_payload=1, output_path=batch_output, accept='application/x-recordio-protobuf')
transformer.transform(data=batch_input, data_type='S3Prefix', content_type='application/x-recordio-protobuf', split_type='RecordIO')
transformer.wait()
```

Using the original corpus as input results in an InternalServerError

Using a subset of the original corpus works so long as it is sufficiently small.

I have tried up to an ml.c4.18xlarge instance, as well as up to 5 instances of ml.c5.4xlarge. I have tried restricting max_payload and/or max_concurrent_transforms to 1. None of these help.

Here is the full error message:
```
ValueError: Error for Transform job lda-2019-06-04-22-56-24-198-2019-06-04-22-56-24-517: Failed Reason: InternalServerError: We encountered an internal error. Please try again.
```

Digging into logs:

the data log only contains the following:
```
MaxConcurrentTransforms=1, MaxPayloadInMB=1, BatchStrategy=MULTI_RECORD
temp-lda-results/lda-d100-predict.data: Unable to get response from algorithm
```

The main log contains, amongst many messages, 5 instances of the following two errors:
```
[CRITICAL] WORKER TIMEOUT (pid:89)
terminate called after throwing an instance of 'std::system_error'. what(): No such process
```

Again, it will run fine with a smaller set of documents, eg 2000 documents are processed without errors, but 20000 will throw this error.

UPDATE: I recently changed the LDA alpha0 parameter from 1 to 0.1. Now the error happens with as few as 500 documents being batch transformed.

Any assistance would be much appreciated.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the shown LDA transformer.transform configuration with the 2,000- and 20,000-document inputs, then compare the data and main logs. Check the reported WORKER TIMEOUT and “Unable to get response from algorithm” messages, including the alpha0 update. Done means batch transform completes for the reported corpus sizes without the InternalServerError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.