MaartenGr / MaartenGr/BERTopic
Can we save generated topic model directly to s3 ?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I have a use-case where I am creating topic model out of given data and want to store to topic model directly to s3 due to memory constraints on the running the service on AWS Lambda.
Any help or idea would help. I tried the pickle approach and directory approach to save the model on s3. Pickle approach throws error and directory approach shows success but doesn't upload anything (show success in Jupyter Notebook, no error thrown)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file or test is named. Start by reproducing the pickle and directory save approaches in the Jupyter Notebook and AWS Lambda context, then determine what behavior is expected for S3 storage; done means the generated topic model uploads successfully and can be used without exceeding the service's memory constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100