Support storing incrementally piped values
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 172
- Avg merge
- 9m
- Merged PRs (30d)
- 1
Description
I'm trying to use sqlite-utils to data generated incrementally. There are a few
aspects of this that I don't currently know how to handle. I would like an option
to apply writes incrementally, line-by-line as they are received. I would like an
option to echo incremental progress. And, it would be nice to have
In particular, I'm using CoreLocationCLI -w -j to generate, newline-delimited JSON.
One variant of the command
stdbuf -oL CoreLocationCLI -w -j | pee 'sqlite-utils insert loc.db loc -' nl
pee, from moreutils, is like tee but spawns and pipes to the processes
created by invoking each of its arguments, so, for gratuitous demonstration,
pee 'sponge out.log' cat would behave like tee.
It looks like I can get what I want with:
stdbuf -oL CoreLocationCLI -w -j | while read line; do <<<"$line" sqlite-utils insert loc.db loc -; echo "$line"; done | nl
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading and running the sqlite-utils insert command with the newline-delimited JSON pipeline described in the issue. Define the desired command options and verify that input is written incrementally and progress can be echoed while the stream is still arriving.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- cli, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100