Proposal: out-of-tree encryption backends

This proposal explores ways to enable out-of-tree encryption backends in SOPS. The goal is to make it easy to add new encryption backends, including private ones.

Other components that could benefit from out-of-tree extensibility, such as stores and publishing backends, are out of scope.

Related PRs:

• #683
• #1097
• #1718
• #1226
• #1906
• #2001
• #1984

As stated multiple times, maintainers are open to a plugin system for encryption backends. This avoids adding dependencies and responsibilities to core SOPS. In the future, moving even core providers out-of-tree would make evolving (and forking) them a lighter process.

I've explored two approaches, and I'm looking for maintainer feedback before investing more time.

Option 1: build on the existing key service infrastructure

This approach was discussed in #481, but I don't think it went anywhere.

I Hacked together sops-generic-keyservice, a proof-of-concept key service server compatible with Scaleway Key Manager and OVH Key Management Service. For a lack of better alternative, it currently piggybacks on AWS' --kms flag to pass arbitrary configuration to the KMS.

I believe this approach can get us far without requiring major/breaking changes in SOPS. Some backward-compatible improvements are needed to make "Generic KMS" a first class backend:

1. Add support for generic_key in the key service API:

   message Key {
     oneof key_type {
       ...
       GenericKey generic_key = 7; // or google.protobuf.Struct ?
     }
   }
   
   message GenericKey {
     // Allow key service servers to filter only the requests they can handle.
     string kms_id = 1;
   
     map<string, google.protobuf.Value> configuration = 2;
   }
   

2. Add the corresponding flags, e.g --generic-kms '{"kms_id": "mykms", "configuration": {"foo": "bar"}}'. Passing a JSON object is inelegant but straightforward. One can also look at how Terraform handles complex dynamic values through flags and environment variables.
3. Add the corresponding configuration options

   creation_rules:
     - generic_kms:
         - kms_id: mykms
           configuration:
             foo: bar
   

Option 2: build a plugin system

A fully fledged plugin system:

• Offers a sleeker user experience, without the need to start services and export environment variables before running SOPS.
• Offers a cleaner API than the keyservice one, purpose-built for providers to implement.
• Lays the groundwork to make other components pluggable, like stores and publishing backends.
• Requires more upfront design, and more coding effort.

This option is not exclusive with option 1: we may rely on a generic key service to interate on the KMS plugin API without affecting core SOPS. Once the API is stable enough, and providers exist for multiple backends (including core ones), we can merge it into SOPS.

I did not invest time in pursuing that option. Maybe PluginRPC would be a good fit, but it leaves installing the executable up to the user.

Next Steps

I'd appreciate feedback from maintainers on pursuing option 1. I can work on a draft PR if this direction sounds reasonable.

Start by reviewing the existing key service API and KMS flag and configuration paths described in the issue, then compare the generic key service and plugin-system options. No repository files or tests are named; done would require maintainer agreement on a direction and a scoped design or draft PR.