INI and DotEnv stores are not roundtrip-safe, and quoting is generally a problem for these formats
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.1k
- Forks
- 1.1k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 9
Description
While looking at #784 I noticed that the situation for both the INI and DotEnv stores is a lot worse than I thought: since they both escape newlines as literal \ns, but do not escape backslashes themselves, they cannot distinguish newlines from \n in plaintext, and both end up as newlines after decrypting.
DotEnv has its own set of problems; there used to be a PR which improved the parser and emitter, see #622, but that had to be reverted since it introduced a breaking change - see #706. The discussions in these PRs, especially the latter, also show that changes to the store formats are dangerous and should be avoided if possible.
This brings up a question on how to handle this mess. The only real fix is to do a breaking change. There are three breaking behaviors:
- When loading an encrypted file written by an older version of sops. This can be handled since sops's version is included in the metadata (as
sops.version). I think this is the very basic thing that any fix must do. This will complicate our code, though. - When emitting plaintext DotEnv or INI files. This can happen while decrypting DotEnv and INI files, but also when decrypting other files (
--output-typeparameter set toiniordotenv). This is definitely a breaking change. - When encrypting a DotEnv or INI file which uses
\nin the input, or some other form of quoting that's suddenly supported.
Especially for 2. and 3. it is probably best if the behavior is configurable. The big question is how can it be configured. For encrypting, .sops.yaml could be used, or command line flags. For decryption, only command line flags and file metadata can be used.
What do folks think about this? I don't expect that we can find a good solution quickly, but we definitely have to start a more focussed discussion :)
CC @getsops/maintainers
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the discussion in issues #784, #622, and #706 to understand the existing INI and DotEnv parsing and emission behavior. The work is done when the project agrees on and implements a safe strategy for round trips, older sops versions, and configurable breaking behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100