Split conversations with IJSON.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
More of a suggestion than an issue, but here's a gist describing what I did with the split_conversations.py script.
https://gist.github.com/itsPat/a1bb06f1dfbd9d4b07288df3bb18b802
Improved it by making use of ijson, so that the input data is processed line by line and then immediately written to the output file instead of storing all the objects in memory.
This will help those with low memory work on large datasets and makes it so the memory required does not scale with the size of the input dataset.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating and reading split_conversations.py, then compare its current processing with the linked gist. Done means the script processes input incrementally and writes output without storing all objects in memory; verify it still produces the expected split conversation files and handles large datasets with lower memory use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100