The Path to Packer Plugin SDK v1.0.0

## The Path to Packer Plugin SDK v1.0.0

The plugin architecture for Packer has proven successful since its inception but has introduced hurdles while working with HCL2 support.

In a recent change to github.com/zclconf/go-cty, a critical package for working with HCL2 types, a breaking change presented issues for Packer HCL2 development - refer to [https://github.com/hashicorp/packer-plugin-sdk#135](https://github.com/hashicorp/packer-plugin-sdk/issues/135) for details.
The change to remove `encoding/gob` from zclconf/go-cty, which was an experiment for Terraform, made it impossible to upgrade Packer because of Packer's reliance on the `encoding/gob` serialization. The use of gob for HCL2 is a decision made early on to avoid major refactors to the Packer plugin architecture.

Looking into other plugin system implementations used within HashiCorp products such as Terraform, Nomad, and Waypoint, we found the standard based on hashicorp/go-plugin, gRPC, and msgpack - a native serialization format for zclconf/go-cty.

This series of proposals below list out several sequential changes for migrating from the bespoke plugin system in Packer from RPC with `encoding/gob` to using hashicorp/go-plugin with gRPC and msgpack.

The solution space for changing Packer's wire protocol involves temporarily patching the breaking change preventing the use of zclconf/go-cty v1.11.0+, augmenting the Packer SDC command to help plugin developers automate the transition,
the migration to hashicorp/go-plugin, and finally the implementation of a gRPC based plugin client/server to be used by Packer core and its plugins.

Logical workstreams are divided into development phases. Discussion of these topics can take place inside each pull-request.

## Phase 1: Packer Plugin SDK 0.5.0 ✅ ([Release Announcement v0.5.1](https://discuss.hashicorp.com/t/changes-to-the-packer-plugin-sdk-v0-5-1/56751))
The first phase resolves the `encoding/gob` issue blocking Packer and the Packer SDK from using updated versions of github.com/zclconf/go-cty and github.com/hashicorp/hcl/v2.

1. Fork github.com/zclconf/go-cty for Packer
1. Add Packer SDK changes required for using the nywilken/go-cty fork.
1. Add Packer core and plugins changes required for using the nywilken/go-cty fork.
1. Introduce `packer-sdc fix` sub-command for automating go.mod file changes to add/remove SDK dependencies.

## Phase 2: Packer Plugin SDK 0.6.0
In the second phase the focus is on creating an adapter around Packer's dependency on gob to slowly migrate Packer's plugin system architecture to a more modern implementation.

1. Update Packer Plugin SDK to support the use of Protobuf/msgpack as the wire protocol for communicating with plugins.
1. Update Packer to toggle between the use of Gob or Protobuf/msgpack for all its over-the-wire communication for handling HCL configurations. All communicators will continue to rely on Gob as the wire protocol.

## Phase 3: Packer Plugin SDK 1.0.0
The exact timeline for this phase will depend on feedback and Packer telemetry data from users.

## Phase 4: Deprecate RPC communication for Packer plugins
Update Packer Plugin SDK to default to Protobufs/msgpack for all plugin communication.