getsops / getsops/sops

INI and DotEnv stores are not roundtrip-safe, and quoting is generally a problem for these formats

Open
#1,435 7 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/stores stores/dotenv stores/ini
Dominant language
Go
Stars
23.1k
Forks
1.1k
Avg merge
1d 11h
Merged PRs (30d)
9

Description

While looking at #784 I noticed that the situation for both the INI and DotEnv stores is a lot worse than I thought: since they both escape newlines as literal \ns, but do not escape backslashes themselves, they cannot distinguish newlines from \n in plaintext, and both end up as newlines after decrypting.

DotEnv has its own set of problems; there used to be a PR which improved the parser and emitter, see #622, but that had to be reverted since it introduced a breaking change - see #706. The discussions in these PRs, especially the latter, also show that changes to the store formats are dangerous and should be avoided if possible.

This brings up a question on how to handle this mess. The only real fix is to do a breaking change. There are three breaking behaviors:

  1. When loading an encrypted file written by an older version of sops. This can be handled since sops's version is included in the metadata (as sops.version). I think this is the very basic thing that any fix must do. This will complicate our code, though.
  2. When emitting plaintext DotEnv or INI files. This can happen while decrypting DotEnv and INI files, but also when decrypting other files (--output-type parameter set to ini or dotenv). This is definitely a breaking change.
  3. When encrypting a DotEnv or INI file which uses \n in the input, or some other form of quoting that's suddenly supported.

Especially for 2. and 3. it is probably best if the behavior is configurable. The big question is how can it be configured. For encrypting, .sops.yaml could be used, or command line flags. For decryption, only command line flags and file metadata can be used.

What do folks think about this? I don't expect that we can find a good solution quickly, but we definitely have to start a more focussed discussion :)

CC @getsops/maintainers

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the discussion in issues #784, #622, and #706 to understand the existing INI and DotEnv parsing and emission behavior. The work is done when the project agrees on and implements a safe strategy for round trips, older sops versions, and configurable breaking behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.